The xz Utils Backdoor of 2024: Caught by a Half-Second Delay

On 29 March 2024, a software engineer posted a short message to a security mailing list. He had found a backdoor in xz, a compression tool and library present on almost every Linux system. It had been placed there deliberately, by someone who had spent roughly two years earning the trust of the project. The discovery began with something tiny: SSH logins that were taking about half a second longer than they should.

How it was found

The engineer was Andres Freund, a developer who works on the PostgreSQL database and who was then employed by Microsoft. While running tests on a Debian development system, he noticed that logins using SSH, the standard way to administer Linux servers remotely, used more processor time than expected and were slightly slower. Most people would have ignored it. Freund investigated and traced the behaviour to the liblzma library, part of the xz package, in versions 5.6.0 and 5.6.1.

What the backdoor did

The malicious code, tracked as CVE-2024-3094, was hidden in the release packages of xz and was not visible in the project's public source repository in the same way. It was inserted through obfuscated build scripts and test files that disguised binary data. On certain systems, it altered how the SSH server verified logins, so that someone holding a specific private key could have run commands remotely. In plain terms, it would have given its author a way into any affected server exposed to the internet.

The conditions for the attack were specific. The SSH daemon on some distributions, including Debian and Fedora, is patched to load a library called libsystemd, which in turn loads liblzma. That chain is how the backdoor ended up inside the process that handles SSH logins. Systems that did not use this chain were not exposed.

The long game

The more troubling part of the story is the way the attacker gained influence. The xz project was maintained largely by one person, Lasse Collin, who had been working on it for years without payment. An account under the name Jia Tan began contributing to the project around 2021. Over time, that account made helpful contributions and gained more responsibility, and in 2023 became a co-maintainer with permission to publish releases.

In 2022, before that promotion, other accounts appeared on the project's mailing list complaining about slow progress and pressing the maintainer to hand over more control. Collin had told the community that he was struggling and that he needed help. Researchers who studied the timeline concluded that the pressure and the offer of help were probably part of a coordinated plan, although the real identity of the person or group behind the accounts has not been publicly established. That uncertainty is worth stating plainly.

Luck played a part

The backdoor was found before it reached the stable releases of the major distributions. It had been included in some development and rolling-release versions, such as Fedora Rawhide, Debian testing and unstable, and a few others, and the affected packages were quickly withdrawn or rolled back. If the discovery had come a few weeks later, the infected versions might have been installed on millions of production servers.

Lessons for open source

• Maintainers are a target. Overworked volunteers can be pressured, and attackers can aim for the people, not the code.
• Source and build can differ. What you read in a repository is not always what ends up in a release package.
• Odd behaviour matters. A small, unexplained change in performance was the thread that unravelled the whole thing.
• Support what you depend on. Many projects underpin commercial services with very little funding.

How to think about updates after xz

It might seem that the lesson is to stop updating. It is not. The backdoor affected the newest versions, and most servers running the stable releases of mainstream distributions were never exposed. The better advice is to use updates from the stable channels of your operating system, which are tested for a period before they reach you, and to avoid pulling the newest upstream releases onto production machines on the day they appear.

It also helps to separate roles. Keep a test or staging environment where updates are applied first, and move them to production once they have been running without trouble. On a single small server, even a short delay of a few days before non-urgent updates can provide a buffer, while security fixes should still go in quickly.

The practical lesson for site owners

Keep your servers updated through your distribution's official channels, and avoid running the very newest development versions of packages on production machines. Restrict SSH to the addresses and users that need it, use key-based logins and disable password logins. Watch your servers for unexplained changes in speed or processor use, since those can be early signs of trouble.

A VPS gives you full control of these settings, and our security page and knowledge base explain how to lock down remote access.