If you’re anything like me, you love a heist film — the plotting, scheming, planning, having to execute everything at an incredibly high level to pull it off and not get caught… it’s great. What’s not great? When the heist is your business, your data, or maybe even worse, the entire internet. That’s how big the xz Utils backdoor hack almost was.

“This might be the best executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library,” software and cryptography engineer Filippo Valsorda said.

Yes, a heist of that size and scale almost succeeded. It was years in the making, and an incredibly observant engineer at Microsoft (with a low bar for inconvenience) might have been the only person or thing that stopped it. So, here’s what it was, how it happened and what you can learn from it:

What is xz Utils?

xz Utils is an essential tool for data compression across Unix-like systems, including Linux. Its ability to handle the legacy .lzma format makes it indispensable for various operations involving compressing and decompressing data.

The Discovery

Andres Freund, working on Microsoft’s PostgreSQL for Debian, was troubleshooting SSH performance issues when he discovered anomalies linked to recent xz Utils updates. Specifically, SSH logins were consuming too many CPU cycles and were generating errors with valgrind, a utility for monitoring computer memory. These updates, revealed on the Open Source Security List, were the result of an intentional backdoor.

The Backdoor’s Function

The backdoor, embedded in versions 5.6.0 and 5.6.1, allowed malicious manipulation of SSH connections. It used a predetermined encryption key to execute any uploaded code during SSH logins, potentially enabling theft of encryption keys or malware installation.

Timeline and Social Engineering

No need to reinvent the wheel here, I’ll just quote from the Wired article:

It would appear that this backdoor was years in the making. In 2021, someone with the username JiaT75 made their first known commit to an open source project. In retrospect, the change to the libarchive project is suspicious, because it replaced the safe_fprint funcion with a variant that has long been recognized as less secure. No one noticed at the time.

The following year, JiaT75 submitted a patch over the XZ Utils mailing list, and, almost immediately, a never-before-seen participant named Jigar Kumar joined the discussion and argued that Lasse Collin, the longtime maintainer of XZ Utils, hadn’t been updating the software often or fast enough. Kumar, with the support of Dennis Ens and several other people who had never had a presence on the list, pressured Collin to bring on an additional developer to maintain the project.

In another Wired article, they make the case that part of what made “Jia Tan” so nearly successful was being exceedingly polite and a willing volunteer contributor to a part of this open code library that was maintained by a single person (Lasse Collin, mentioned above). They spent years building up trust and goodwill within the community so that when it was time to plant the backdoor, they had the standing to get away with it.

This type of social engineering is often where big breaches occur (and why we always caution business owners that every system is only as secure as the people with access to it).

Now, back to the specifics from the first article:

In January 2023, JiaT75 made their first commit to XZ Utils. In the months following, JiaT75, who used the name Jia Tan, became increasingly involved in XZ Utils affairs. For instance, Tan replaced Collins’ contact information with their own on oss-fuzz, a project that scans open source software for vulnerabilities that can be exploited. Tan also requested that oss-fuzz disable the ifunc function during testing, a change that prevented it from detecting the malicious changes Tan would soon make to XZ Utils.

In February of this year, Tan issued commits for versions 5.6.0 and 5.6.1 of XZ Utils. The updates implemented the backdoor. In the following weeks, Tan or others appealed to developers of Ubuntu, Red Hat, and Debian to merge the updates into their OSes. Eventually, one of the two updates made its way into several releases, according to security firm Tenable. There’s more about Tan and the timeline here.

Backdoor Mechanics

The backdoor allowed attackers with the correct key to hijack SSH connections, executing commands hidden within certificates. It used a five-stage loader with sophisticated techniques to remain concealed.

What You Can Learn From This

I realize much of that is pretty dang technical. The main takeaways, though, are that social engineering often comes in the form of friendly help. And, given how interconnected and complex our technology and systems have become, it doesn’t take a big back door to compromise massive swathes of our computers, systems and ultimately, lives.

That’s why we’re so passionate about being a Managed Service Provider — we know very few small and medium-sized businesses have the time, resources or expertise to monitor and protect themselves from threats like this on their own. We love being our clients’ trusted partner, helping maintain their security posture (and running tests and training to help improve staff vulnerability to things like social engineering of this nature).

If you could use a hand protecting your business from threats like these, give us a call — we’d love to chat.