On March 29, Microsoft software developer Andres Freund was trying to optimize the performance of his computer when he noticed that one program was using an unexpected amount of processing power. Freund dove in to troubleshoot and “got suspicious.”
Eventually, Freund found the source of the problem, which he subsequently posted to a security mailing list: He had discovered a backdoor in XZ Utils, a data compression utility used by a wide array of various Linux-based computer applications — a constellation of open-source software that, while often not consumer-facing, undergirds key computing and internet functions like secure communications between machines.
By inadvertently spotting the backdoor, which was buried deep in the code in binary test files, Freund averted a large-scale security catastrophe. Any machine running an operating system that included the backdoored utility and met the specifications laid out in the malicious code would have been vulnerable to compromise, allowing an attacker to potentially take control of the system.
The XZ backdoor was introduced by way of what is known as a software supply chain attack, which the National Counterintelligence and Security Center defines as “deliberate acts directed against the supply chains of software products themselves.” The attacks often employ complex ways of changing the source code of the programs, such as gaining unauthorized access to a developer’s system or through a malicious insider with legitimate access.
The malicious code in XZ Utils was introduced by a user calling themself Jia Tan, employing the handle JiaT75, according to Ars Technica and Wired. Tan had been a contributor to the XZ project since at least late 2021 and built trust with the community of developers working on it. Eventually, though the exact timeline is unclear, Tan ascended to being co-maintainer of the project, alongside the founder, Lasse Collin, allowing Tan to add code without needing the contributions to be approved. (Neither Tan nor Collin responded to requests for comment.)
The XZ backdoor betrays a sophisticated, meticulous operation. First, whoever led the attack identified a piece of software that would be embedded in a vast array of Linux operating systems. The development of this widely used technical utility was understaffed, with a single, core maintainer, Collin, who later conceded he was unable to maintain XZ, providing the opportunity for another developer to step in. Then, after cultivating Collin’s trust over a period of years, Tan injected a backdoor into the utility. All these moves were underlaid by a technical proficiency that ushered the creation and embedding of the actual backdoor code — a code sophisticated enough that analysis of its precise functionality and capability is still ongoing.
“The care taken to hide the exploits in binary test files as well as the sheer time taken to gain a reputation in the open-source project to later exploit it are abnormally sophisticated,” said Molly, a system administrator at Electronic Frontier Foundation who goes by a mononym. “However, there isn’t any indication yet whether this was state sponsored, a hacking group, a rogue developer, or any combination of the above.”
Tan’s elevation to being a co-maintainer mostly played out on an email group where code developers — in the open-source, collaborative spirit of the Linux family of operating systems — exchange ideas and strategize to build applications.
On one email list, Collin faced a raft of complaints. A group of users, relatively new to the project, had protested that Collin was falling behind and not making updates to the software quickly enough. He should, some of these users said, hand over control of the project; some explicitly called for the addition of another maintainer. Conceding that he could no longer devote enough attention to the project, Collin made Tan a co-maintainer.
The users involved in the complaints seemed to materialize from nowhere — posting their messages from what appear to be recently created Proton Mail accounts, then disappearing. Their entire online presence is related to these brief interactions on the mailing list dedicated to XZ; their only recorded interest is in quickly ushering along updates to the software.
Various U.S. intelligence agencies have recently expressed interest in addressing software supply chain attacks. The Cybersecurity and Infrastructure Security Agency jumped into action after Freund’s discovery, publishing an alert about the XZ backdoor on March 29, the same day Freund publicly posted about it.
Open-Source Players
In the open-source world of Linux programming — and in the development of XZ Utils — collaboration is carried out through email groups and code repositories. Tan posted on the listserv, chatted to Collin, and contributed code changes on the code repository Github, which is owned by Microsoft. GitHub has since disabled access to the XZ repository and disabled Tan’s account. (In February, The Intercept and other digital news firms sued Microsoft and its partner OpenAI for using their journalism without permission or credit.)
Several other figures on the email list participated in efforts — appearing to be diffuse but coinciding in their aims and timing — to install the new co-maintainer, sometimes particularly pushing for Tan.
Later, on a listserv dedicated to Debian, one of the more popular of the Linux family of operating systems, another group of users advocated for the backdoored version of XZ Utils to be included in the operating system’s distribution.
These dedicated groups played discrete roles: In one case, complaining about the lack of progress on XZ Utils and pushing for speedier updates by installing a new co-maintainer; and, in the other case, pushing for updated versions to be quickly and widely distributed.
“I think the multiple green accounts seeming to coordinate on specific goals at key times fits the pattern of using networks of sock accounts for social engineering that we’ve seen all over social media,” said Molly, the EFF system administrator. “It’s very possible that the rogue dev, hacking group, or state sponsor employed this tactic as part of their plan to introduce the back door.
It is also conceivable that these occurrences are mere coincidences.
The behavior observed aligns with what is known in the intelligence community as “persona management,” which involves creating and maintaining multiple fictitious identities. A leaked document from defense contractor HBGary Federal details the intricacies involved in maintaining these fake personas, including establishing a detailed online presence – a component noticeably absent from the accounts in the XZ timeline.
While these users used different email addresses, some utilized providers that offer clues about when their accounts were established. For instance, when using Proton Mail accounts, the encryption keys linked to these accounts were generated on the same day or shortly before the users’ initial posts on the email group. (It is worth noting that users can generate new keys, potentially indicating that the email addresses might have been older than their current keys.)
One of the early users on the list was identified as Jigar Kumar. Kumar made an appearance on the XZ development mailing list in April 2022, expressing confusion about certain features of the tool. Tan promptly responded to Kumar’s comment. (Kumar did not provide a response when contacted.)
Kumar continued to surface with subsequent complaints, sometimes piggybacking off others’ dissatisfaction. Following Dennis Ens’ appearance on the same mailing list, Ens also voiced concerns about lack of response to one of his messages. Collin acknowledged the mounting issues and mentioned that Tan had been assisting him off-list; Tan may soon play a more prominent role with XZ Utils. (Ens did not respond when contacted.)
After another complaint from Kumar calling for a new maintainer, Collin revealed his challenges in maintaining interest due to long-term mental health issues. He mentioned working with Jia Tan off-list and hinted at Tan potentially assuming a larger role in the future.
The pressure continued to escalate, with Collin hinting at Tan’s increased involvement in the project. Ens, who had been silent for two years, reappeared around the time when the malicious backdoor code was introduced into the XZ software, advocating for quicker updates.
After Tan was appointed as a co-maintainer, there was a concerted effort to widely distribute XZ Utils, now containing the backdoor. Hans Jansen emerged on the scene, pushing for the new version of XZ to be included in Debian Linux. An individual at Red Hat recounted Tan’s attempt to persuade him to incorporate the compromised XZ Utils into Fedora.
These widely-used Linux operating systems cater to millions of users, implying that a significant number of users were at risk of compromise if Freund had not uncovered the backdoor.
Molly emphasized that while the potential for socially engineering backdoors in critical software may raise concerns about open-source projects, it is not limited to open source and could occur in any environment. The timely discovery of the backdoor by the engineer was only possible due to the transparency of the project.
