Progress Linux

A backdoor in xz-utils (CVE-2024-3094)

Andres Freund discovered on 2024-03-29 that the upstream source tarballs for xz-utils, the XZ-format compression utilities, are compromised and inject malicious code, at build time, into the resulting liblzma5 library (supply chain attack).

Right now no Progress Linux stable versions are known to be affected. Compromised packages were part of the graograman-backports distribution, with versions ranging from 5.6.0-0.1~progress7+u1 (uploaded on 2024-02-26), up to and including 5.6.1-1~progress7+u1. The package has been reverted on 2024-03-30 to use the upstream 5.4.5 code, which is versioned as 5.6.1+really5.4.5-1~progress7+u1.

Users running Progress Linux 7.99 (graograman-backports) are urged to update the xz-utils packages.

Both the Git server as well as the hidden-primary repository server for Progress Linux were running on graograman-backports with affected versions of xz-utils (indirect depends due to kernel related backports). After an audit of all Git repositories and Debian packages in our archive, we have not found any signs of compromise.

As far as the backdoor has been analyzed by the community, it allows for remote execute arbitrary code on affected systems. We have secured our systems and have not found any signs of a compromise, but as one of the repository signing keys was (passphrase protected) on one of the servers, we treat all data and systems as compromised and will clean-room re-setup the entire infrastructure, replace all PGP and SSH keys, regenerate all Git repositories and rebuild all Debian packages. This is the only way to be completely sure about the integrity of the project.

For more information about the xz backdoor, see:

Timeline

  • 2024-03-29: Backdoor in xz has been discovered.

  • 2024-03-30: Progress Linux infrastructure secured with xz-utils 5.6.1+really5.4.5-1~progress7+u1.

  • 2024-03-30: All systems locked down and archive processing stopped.

  • 2024-03-30: Audit of all affected systems, all Git repositories and all Debian packages found no signs of compromise.

  • 2024-03-31: Setup air-gapped system on new hardware for bootstrapping.

  • 2024-03-31: Moved all containers from server #1 to server #2.

  • 2024-03-31: Wiping server #1.

  • 2024-04-01: Generated new PGP and SSH keys on air-gapped system.

  • 2024-04-01: Setup server #1.

  • 2024-04-01: Wiping buildd #1.

  • 2024-04-02: Setup amd64 buildd #1.

  • 2024-04-06: Setup temporary Git server.

  • 2024-04-07: Setup temporary primary package repository server.

  • 2024-04-08: Rebuild of stable repository is completed.

TODO

  • Add second Debian keyring signature to archive signing keys.

  • Document key rollover instructions.

  • Resetup secondary repository servers.

  • Regenerate stable-extras, stable-backports and stable-backports-extras Git repositories.

  • Rebuild all Debian amd64 packages.

  • Resetup remaining servers.

  • Resetup remaining buildds.

  • Resetup remaining containers.

  • Upload new archive-keys to Debian unstable.

  • Rebuild packages for remaining architectures (arm64 etc.).

  • Regenerate oldstable and oldoldstable repositories.

  • Rebuild all oldstable and oldoldstable packages.

  • Upload new archive-keys to Debian via SRM in stable, oldstable, and old-oldstable.

  • Find a sponsor for 2 HSMs to store the signing keys on the primary repository server.