<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=705633339897683&amp;ev=PageView&amp;noscript=1">

 

The xz backdoor: What you need to know

News, perspectives, and recommendations for application development teams dealing with the xz utils backdoor hack

xz utils hack: what is it?

In late March 2024, a developer noticed some unusual behavior on their computer, investigated it,  and uncovered a hack of epic scope, in an obscure but important library called xz. The attack was technically sophisticated, but perhaps worse, it was socially sophisticated. The attackers took advantage of an open source maintainer over a long period of time to slowly, but steadily, win his trust—and then subvert the security mechanisms that he had previously put in place.

The maintainer facing this deliberate, long-term attack was, in his own words at the time the hack began, “unpaid.”

“I haven’t lost interest but my ability to care has been fairly limited... it’s also good to keep in mind that this is an unpaid hobby project.”

In the same email, this maintainer said:

“Jia Tan may have a bigger role in the project in the future. He has been helping a lot off-list and is practically a co-maintainer already. :-)”

Like any good horror story, you can see where this is going: it was exactly this “Jia Tan” who, over a period of two years, took over xz and inserted a malicious backdoor that could have exposed computers the world over to remote execution.

We got lucky this time: the problem was caught early. But it will not be the last time this sort of sophisticated attack is tried.

Read the full recap

What does the xz hack mean for organizations using open source?

The Internet has blossomed with takes on the xz hack, and we share many of the best ones below. Many of them were a mix of angry and sad reactions that this maintainer, clearly operating in good faith, had been taken advantage of. Past that, though, there was a lot of disagreement about how to reduce the odds of this happening in the future. For organizations heavily reliant on open source in their applications (which is most or all of them), the xz hack is a wakeup call. It is a warning sign that they need to become increasingly vigilant in ensuring that the open source they are using is secure, and that the people who create and maintain the software they use are properly supported in their work.

One of those frequently recurring themes in commentary about the hack: “how can we work together to pay maintainers like this one, so they are more resilient to attacks like this one?”

Read our maintainer impact report

What does the xz hack mean for open source maintainers?

Tidelift’s 2023 state of the open source maintainer report found that 60% of open source maintainers describe themselves as unpaid hobbyists, while only 13% report earning most or all of their income from maintaining their projects.

Meanwhile 44% of maintainers describe themselves—like the xz maintainer—as solo maintainers. So it is no surprise that, when asked what they dislike most about being a maintainer, they reported that it is stressful, lonely, demanding, and financially unrewarding work. In fact, almost 60% of maintainers have either quit or considered quitting maintaining their projects.

The xz hack brings the reality of life as a maintainer into stark relief, and hopefully generates real, meaningful change to protect the technology infrastructure we all rely on.

Read the full survey results

Tidelift survey figures for blog v.6_Figure 1 60_ unpaid

 

cves-chart

The number of CVEs published in the National Vulnerability Database has increased rapidly since 2017. If less vulnerable, more secure software is the goal, should we consider a different approach? Tidelift VP of Product Lauren Hanford shares her thoughts.

Read the post

 

How can Tidelift—and our customers—help?

Tidelift is improving the security and resilience of the open source software supply chain, specifically focused on the most depended upon application development libraries in popular ecosystems like JavaScript, Java, Python, Ruby, and Go.

Tidelift is the only company that partners with open source maintainers and pays them to:

  • Implement industry-leading secure software development practices and validate the practices they follow so organizations can have the same confidence in the security of their open source that they have in their own code.
  • Contractually commit to continue these practices into the future so that organizations can confidently make long term investments in the packages they use.

While we do not cover C/C++ packages like xz today, we can help organizations prepare for issues like this in the future across our supported ecosystems. Here are some of the ways Tidelift can help:

  • We pay maintainers to reach security and licensing goals. Put simply, we pay maintainers to help them become better maintainers, and to keep maintaining the software.
  • We pay known maintainers and work to ensure that the money goes to the people already doing the maintenance.
  • We pay based on project usage. We pay maintainers based on an analysis of how many of our customers use their software.
  • When a maintainer partner needs to move on, we help ensure project continuity. Sometimes, for perfectly good reasons, a maintainer wants to stop working on their project. When this happens, we help find trusted maintainers from our network interested in getting paid to continue the work.
  • We focus on the middle of the stack. The traditional operating systems vendors have built a lot of infrastructure to vet and support core operating system features, and the Linux Foundation and others have put in a lot of work around the biggest high-level development tools and frameworks. But that has left a huge number of relatively untouched packages in every modern language stack that need Tidelift-style attention. We believe our highest impact, on open source and for our customers, lies in supporting those packages.

Catch the Upstream session on-demand!

Panel: Life as a maintainer after the xz utils backdoor hack

The explosive details about this xz utils backdoor hack, in which a volunteer open source maintainer was manipulated over a period of years into giving commit access to their project, haa sent shudders across all open source communities.

But it was particularly scary for open source maintainers, who now have new vectors they have to consider in their work around trust in those they collaborate with, often over the Internet without ever having met face to face.

We spoke with five prominent maintainers working in the Javascript, Java, and Python ecosystems to hear directly from them about what life as a maintainer will look like in a post xz world.

Watch the panel

Screenshot 2024-04-12 at 3.58.07 PM

 

1200x628 (5)

 

Where to learn more about the xz hack

Here are some of the articles that we recommend as basic reading to fully understand how the xz hack happened and how it was uncovered:

Frequently asked questions

What is xz utils?

xz utils (which used to be called LZMA Utils) is a set of free, open source command-line lossless data compressors, which includes lzma and xz, for Unix-like operating systems and, from version 5.0 onwards, Microsoft Windows. 

xz utils is commonly used for compressing release tarballs, software packages, kernel images, and initramfs images. Because it's so widely used, statistically your average Linux or macOS system will have it installed for convenience.

This means, had the hack succeeded,...

xz utils (which used to be called LZMA Utils) is a set of free, open source command-line lossless data compressors, which includes lzma and xz, for Unix-like operating systems and, from version 5.0 onwards, Microsoft Windows. 

xz utils is commonly used for compressing release tarballs, software packages, kernel images, and initramfs images. Because it's so widely used, statistically your average Linux or macOS system will have it installed for convenience.

This means, had the hack succeeded, millions of computers could have been compromised.

What is the xz backdoor hack?

On March 29, a Microsoft developer was trying to optimize his computer when he noticed that one program was using an unexpected amount of processing power. 

After diving into the problem, he realized the source of the problem, which he subsequently posted on a security mailing list.

"After observing a few odd symptoms around liblzma (part of the xz package) onDebian sid installations over the last weeks (logins with ssh taking a lot ofCPU, valgrind errors) I figured out the answer:"The upstream...

On March 29, a Microsoft developer was trying to optimize his computer when he noticed that one program was using an unexpected amount of processing power. 

After diving into the problem, he realized the source of the problem, which he subsequently posted on a security mailing list.

"After observing a few odd symptoms around liblzma (part of the xz package) on
Debian sid installations over the last weeks (logins with ssh taking a lot of
CPU, valgrind errors) I figured out the answer:

"The upstream xz repository and the xz tarballs have been backdoored.

"At first I thought this was a compromise of debian's package, but it turns out
to be upstream."

This discovery sent the internet into a tizzy.

Why is the xz backdoor hack important?

Any machine running an operating system that included the backdoored utility and met the specifications outlined in the malicious code could be at risk of compromise, potentially enabling an attacker to gain control of the system.

This attack was especially vicious, because the bad actor who inserted the malicious code did so by gaining the trust of an overworked, underpaid maintainer over years

This wasn't just a technically efficient attack; it was socially sophisticated in a way that...

Any machine running an operating system that included the backdoored utility and met the specifications outlined in the malicious code could be at risk of compromise, potentially enabling an attacker to gain control of the system.

This attack was especially vicious, because the bad actor who inserted the malicious code did so by gaining the trust of an overworked, underpaid maintainer over years

This wasn't just a technically efficient attack; it was socially sophisticated in a way that should ring alarm bells across any organization using open source—and should rally them into doing something to help protect maintainers!

Who is Jia Tan?
No one knows for sure who the mysterious Jia Tan is, as it appears to be a fake name. Jia Tan is the contributor who took over for the maintainer of xz after gaining his trust for years in a coordinated social scheme.
No one knows for sure who the mysterious Jia Tan is, as it appears to be a fake name. Jia Tan is the contributor who took over for the maintainer of xz after gaining his trust for years in a coordinated social scheme.
What is Linux xz? What is xz/liblzma?
Linux xz and xz/liblzma are old names for the library known now as xz utils. XZ utils is a data compression library available for the Linux operating system.
Linux xz and xz/liblzma are old names for the library known now as xz utils. XZ utils is a data compression library available for the Linux operating system.
How to reduce the threat of hacks like xz?
Tidelift can help your organization reduce the threat of hacks like xz by giving you visibility into the open source libraries that make up 70% of your applications.
Tidelift can help your organization reduce the threat of hacks like xz by giving you visibility into the open source libraries that make up 70% of your applications.

Resources for your organization:

10-questions

 

10 questions you should answer before using a new open source project

When it comes to open source software security, many organizations rely heavily on software scanning (often called software composition analysis or SCA) as the primary means of defense.

While scanning helps protect against known vulnerabilities reactively, leading organizations today are adding proactive defenses that help them make better decisions about which open source packages to bring into their supply chain in the first place.

Read the infographic

Who's gonna do the work?

In a webinar we hosted last year detailing the government's reaction to another infamous open source software vulnerability, Log4Shell, Tidelift CEO Donald Fischer asks the pressing question: who is going to do all the work insuring open source meets industry standards?

Watch now

Screenshot 2024-04-04 at 11.35.42 AM

 

Employers  + Tidelift

 

“[The Tidelift maintainer] relationship is pure gold. The openness you have with the open source maintainers and the ability to talk with the consumers about how we’re using their products—we have a direct line of communication from their fixes and what versions we should be using.” 

Read how Employers' insurance is streamlining workflows and lowering research costs while also reducing open source software related risk.

Read now

Maintainer burnout is real. Almost 60% of maintainers have quit or considered quitting maintaining one of their projects.

Fifty-eight percent of maintainers have either quit (22%) or considered quitting (36%) their maintenance work on a project, which is almost identical to what we found in our previous survey. A minority of maintainers (43%), have not quit or considered quitting maintaining their projects.

Read the survey

Tidelift survey figures for blog v.6_Figure 17 Quitting

 

Learn how one large organization saved over $1.6M in manual package evaluation time and eliminated over 3,000 points of risk in applications running in production.

Resources to reduce cybersecurity risk from bad open source packages: