Progress Linux

A backdoor in xz-utils (CVE-2024-3094)

Andres Freund discovered on 2024-03-29 that the upstream source tarballs for xz-utils, the XZ-format compression utilities, are compromised and inject malicious code, at build time, into the resulting liblzma5 library (supply chain attack).

Right now no Progress Linux stable versions are known to be affected. Compromised packages were part of the graograman-backports distribution, with versions ranging from 5.6.0-0.1~progress7+u1 (uploaded on 2024-02-26), up to and including 5.6.1-1~progress7+u1. The package has been reverted on 2024-03-30 to use the upstream 5.4.5 code, which is versioned as 5.6.1+really5.4.5-1~progress7+u1.

Users running Progress Linux 7.99 (graograman-backports) are urged to update the xz-utils packages.

Both the Git server as well as the hidden-primary repository server for Progress Linux were running on graograman-backports with affected versions of xz-utils. After an audit of all Git repositories and Debian packages in our archive, we have not found any signs of compromise.

As far as the backdoor has been analyzed by the community, it allows for remote execute arbitrary code on affected systems. We have secured our systems and have not found any signs of a compromise, but as one of the repository signing keys was (passphrase protected) on one of the servers, we treat all data and systems as compromised and will clean-room resetup the entire infrastructure, replace all PGP and SSH keys, regenerate all Git repositories and rebuild all Debian packages. This is the only way to be completely sure about the integrity of the project.

On 2024-04-22 the new rebuilt package archive is in place, see XZ fixup for instructions.

Timeline

  • 2024-03-29: Backdoor in xz has been discovered.

  • 2024-03-30: Progress Linux infrastructure secured with xz-utils 5.6.1+really5.4.5-1~progress7+u1.

  • 2024-03-30: All systems locked down and archive processing stopped.

  • 2024-03-30: Audit of all affected systems, all Git repositories and all Debian packages found no signs of compromise.

  • 2024-03-31: Setup air-gapped system on new hardware for bootstrapping.

  • 2024-03-31: Moved all containers from server #1 to server #2.

  • 2024-03-31: Wiping server #1.

  • 2024-04-01: Generated new PGP and SSH keys on air-gapped system.

  • 2024-04-01: Setup server #1.

  • 2024-04-01: Wiping amd64 buildd #1.

  • 2024-04-02: Setup amd64 buildd #1.

  • 2024-04-06: Setup temporary Git server.

  • 2024-04-07: Setup temporary primary package repository server (apt.progress-linux.org).

  • 2024-04-08: Rebuild of stable, stable-security and stable-updates repositories for amd64 are completed.

  • 2024-04-20: Rebuild of stable-backports repository for amd64 is completed.

  • 2024-04-20: Setup secondary package repository servers (deb.progress-linux.org).

  • 2024-04-20: Uploaded new progress-linux keyring package to Debian unstable.

  • 2024-04-20: New archive pushed to deb.progress-linux.org signed by new keys.

  • 2024-04-21: Rebuild of stable-extras and stable-backports-extras repositories for amd64 are completed.

  • 2024-04-21: Wiping server #2.

  • 2024-04-21: Setup server #2.

  • 2024-04-21: Wiping arm64 buildd #1.

  • 2024-04-21: Setup arm64 buildd #1.

  • 2024-04-26: Rebuild of stable, stable-security and stable-updates repositories for arm64 are completed.

  • 2024-04-26: Rebuild of stable-extras and stable-backports-extras repositories for arm64 are completed.

TODO

  • Regenerate oldstable and oldoldstable repositories.

  • Rebuild all oldstable and oldoldstable packages.

  • Resetup remaining servers.

  • Resetup remaining buildds.

  • Resetup remaining containers.

  • Rebuild packages for remaining architectures (i386).

  • Upload new archive-keys to Debian via SRM in stable, oldstable, and old-oldstable.

  • Get 2 HSMs to store the signing keys on the primary repository server.