WASHINGTON, April 5 – German software developer Andres Freund was running some detailed performance tests last month when he noticed odd behavior in a little known program. What he found when he investigated has sent shudders across the software world and drawn attention from tech executives and government officials.
Freund, who works for Microsoft out of San Francisco, discovered that the latest version of the open source software program XZ Utils had been deliberately sabotaged by one of its developers, a move that could have carved out a secret door to millions of servers across the internet.
Security experts say it’s only because Freund spotted the change before the latest version of XZ had been widely deployed that the world was spared a digital security crisis.
“We really dodged a bullet,” said Satnam Narang, a security researcher with Tenable who has been tracking the fallout from the find. “It is one of those moments where we have to wipe our brow and say, ‘We were really lucky with this one.’”
The near-miss has refocused attention on the safety of open source software – free, often volunteer-maintained programs whose transparency and flexibility mean they serve as the foundation for the internet economy.
Many such projects depend on a tiny circle of unpaid volunteers fighting to get out from under a pile of demands for fixes and upgrades.
XZ, a suite of file compression tools packaged into distributions of the Linux operating system, was long maintained by a single author, Lasse Collin.
In recent years, he appeared to be under strain.
In a message posted to a public mailing list in June 2022, Collin said he was dealing with “longterm mental health issues” and hinted that he working privately with a new developer named Jia Tan and that “perhaps he will have a bigger role in the future.”
Update logs available through the open source software site Github show that Tan’s role quickly expanded. By 2023 the logs show Tan was merging his code into XZ, a sign that he had won a trusted role in the project.
But cybersecurity experts who’ve scoured the logs say that Tan was masquerading as a helpful volunteer. Over the next few months, they say, Tan introduced a nearly invisible backdoor into XZ.
Collin didn’t return messages seeking comment and said on his website that he would not respond to reporters until he understood the situation well enough to do so.
Tan did not return messages sent to his Gmail account. Reuters has been unable to ascertain who Tan is, where he is, or who he was working for, but many of those who’ve examined his updates believe Tan is a pseudonym for an expert hacker or group of hackers — likely one working on behalf of a powerful intelligence service.
“This is not kindergarten stuff,” said Omkhar Arasaratnam, the general manager of the Open Source Security Foundation, which works to defend projects like XZ. “This is incredibly sophisticated.”
‘WE LUCKED OUT’
Tan could easily have gotten away with it had it not been for Freund, the Microsoft developer, whose curiosity was piqued when he noticed the latest version of XZ intermittently using an unexpected amount of processing power on the system he was testing.
Microsoft declined to make Freund available for an interview, but in a publicly-available email and posts to social media, Freund said a series of easy-to-miss clues prompted him to discover the backdoor.
The find “really required a lot of coincidences,” Freund said on the social network Mastodon.
Microsoft CEO Satya Nadella congratulated Freund over the weekend, saying in a post to the social network X that he loved seeing how the developer, “with his curiosity and craftsmanship, was able to help us all.”
In the open source community, the discovery has been sobering. The volunteers who maintain the software that underpins the internet aren’t strangers to the idea of little pay or recognition, but the realization that they were now being hunted by well-resourced spies pretending to be Good Samaritans was “incredibly intimidating,” said Arasaratnam, of the Open Source Security Foundation.
Government officials are also weighing the implications of the near-miss, which has underlined concerns about how to protect open source software. Assistant National Cyber Director Anjana Rajan told Politico that “there’s a lot of conversations that we need to have about what we do next” to protect open source code.”
The Cybersecurity and Infrastructure Security Agency (CISA) says it has been leaning on U.S. companies that use open source software to plow resources back into the communities that build and maintain it. CISA adviser Jack Cable told Reuters the burden was on tech companies not just to vet open software but to “contribute back and help build the sustainable open source ecosystem that we get so much value from.”
It’s not clear that software companies are properly incentivized to do so. Online open source mailing lists are teeming with complaints about tech giants demanding that volunteers troubleshoot issues with open source software those companies use to make billions of dollars.
Whatever the solution, almost everyone agrees the XZ episode shows something has to change.
“We got unreasonably lucky here,” said Freund in another Mastodon post. “We can’t just bank on that going forward.”
