In late March 2024, a developer noticed some unusual behavior on their computer, investigated it, and uncovered a hack of epic scope, in an obscure but important library called xz. The attack was technically sophisticated, but perhaps worse, it was socially sophisticated. The attackers took advantage of an open source maintainer over a long period of time to slowly, but steadily, win his trust—and then subvert the security mechanisms that he had previously put in place.
The maintainer facing this deliberate, long-term attack was, in his own words at the time the hack began, “unpaid.”
“I haven’t lost interest but my ability to care has been fairly limited... it’s also good to keep in mind that this is an unpaid hobby project.”
In the same email, this maintainer said:
“Jia Tan may have a bigger role in the project in the future. He has been helping a lot off-list and is practically a co-maintainer already. :-)”
Like any good horror story, you can see where this is going: it was exactly this “Jia Tan” who, over a period of two years, took over xz and inserted a malicious backdoor that could have exposed computers the world over to remote execution.
We got lucky this time: the problem was caught early. But it will not be the last time this sort of sophisticated attack is tried.
The Internet has blossomed with takes on the xz hack, and we share many of the best ones below. Many of them were a mix of angry and sad reactions that this maintainer, clearly operating in good faith, had been taken advantage of. Past that, though, there was a lot of disagreement about how to reduce the odds of this happening in the future. For organizations heavily reliant on open source in their applications (which is most or all of them), the xz hack is a wakeup call. It is a warning sign that they need to become increasingly vigilant in ensuring that the open source they are using is secure, and that the people who create and maintain the software they use are properly supported in their work.
One of those frequently recurring themes in commentary about the hack: “how can we work together to pay maintainers like this one, so they are more resilient to attacks like this one?”
Tidelift’s 2023 state of the open source maintainer report found that 60% of open source maintainers describe themselves as unpaid hobbyists, while only 13% report earning most or all of their income from maintaining their projects.
Meanwhile 44% of maintainers describe themselves—like the xz maintainer—as solo maintainers. So it is no surprise that, when asked what they dislike most about being a maintainer, they reported that it is stressful, lonely, demanding, and financially unrewarding work. In fact, almost 60% of maintainers have either quit or considered quitting maintaining their projects.
The xz hack brings the reality of life as a maintainer into stark relief, and hopefully generates real, meaningful change to protect the technology infrastructure we all rely on.
The number of CVEs published in the National Vulnerability Database has increased rapidly since 2017. If less vulnerable, more secure software is the goal, should we consider a different approach? Tidelift VP of Product Lauren Hanford shares her thoughts.
Tidelift is improving the security and resilience of the open source software supply chain, specifically focused on the most depended upon application development libraries in popular ecosystems like JavaScript, Java, Python, Ruby, and Go.
Tidelift is the only company that partners with open source maintainers and pays them to:
While we do not cover C/C++ packages like xz today, we can help organizations prepare for issues like this in the future across our supported ecosystems. Here are some of the ways Tidelift can help:
The explosive details about this xz utils backdoor hack, in which a volunteer open source maintainer was manipulated over a period of years into giving commit access to their project, haa sent shudders across all open source communities.
But it was particularly scary for open source maintainers, who now have new vectors they have to consider in their work around trust in those they collaborate with, often over the Internet without ever having met face to face.
We spoke with five prominent maintainers working in the Javascript, Java, and Python ecosystems to hear directly from them about what life as a maintainer will look like in a post xz world.
.png?width=600&height=315&name=1200x628%20(5).png)
Here are some of the articles that we recommend as basic reading to fully understand how the xz hack happened and how it was uncovered:
When it comes to open source software security, many organizations rely heavily on software scanning (often called software composition analysis or SCA) as the primary means of defense.
While scanning helps protect against known vulnerabilities reactively, leading organizations today are adding proactive defenses that help them make better decisions about which open source packages to bring into their supply chain in the first place.
In a webinar we hosted last year detailing the government's reaction to another infamous open source software vulnerability, Log4Shell, Tidelift CEO Donald Fischer asks the pressing question: who is going to do all the work insuring open source meets industry standards?
“[The Tidelift maintainer] relationship is pure gold. The openness you have with the open source maintainers and the ability to talk with the consumers about how we’re using their products—we have a direct line of communication from their fixes and what versions we should be using.”
Read how Employers' insurance is streamlining workflows and lowering research costs while also reducing open source software related risk.
Fifty-eight percent of maintainers have either quit (22%) or considered quitting (36%) their maintenance work on a project, which is almost identical to what we found in our previous survey. A minority of maintainers (43%), have not quit or considered quitting maintaining their projects.
50 Milk St, 16th Floor, Boston, MA 02109
.svg){kind=icon}