The hack that almost broke the internet

This episode explores a major cybersecurity breach involving the open-source software, XZ, which nearly compromised global digital infrastructure.

1. Critical Open Source Vulnerabilities: The episode illustrates the potential dangers inherent in the open-source software model.
2. Economic Impact of Hacking: It discusses the broad economic implications of cybersecurity vulnerabilities.
3. Community and Trust in Software Development: The narrative underscores the community-driven nature of software development and the trust placed in contributors.
4. The Need for Vigilance: The hack underscores the importance of continuous oversight and updating of software to guard against vulnerabilities.
5. Call for Sustainable Management: There's a pressing need for sustainable management practices within the open-source community.

1: The First Clue

Richard Jones recalls the initial benign email that led to a massive cybersecurity breach. The chapter outlines the discovery and initial handling of the suspicious update. Richard Jones: "I just didn't expect that somebody would try that."

2: The Open Source Dilemma

Discusses the evolution of open-source software and its pivotal role in modern technology, juxtaposed with its vulnerabilities. Bruce Perens: "Back then, it was like the wild west of software."

3: The Discovery

Details the critical moments leading to the discovery of the hack, emphasizing the accidental nature of the find. Omkar Arasaratnam: "It gets broad distribution, whoever GTN is quietly logs into computers all over the Internet."

1. Regularly Update Software: Ensure all software, especially open-source, is regularly updated to mitigate risks.
2. Contribute to Open Source: Engage with the open-source community to improve and secure software.
3. Educate on Cybersecurity: Stay informed about potential vulnerabilities in your digital tools.
4. Support Sustainable Practices: Advocate for and support sustainable management of open-source projects.
5. Implement Robust Security Measures: Employ comprehensive security measures to protect against potential breaches.

Last month, the world narrowly avoided a cyberattack of stunning ambition. The targets were some of the most important computers on the planet. Computers that power the internet. Computers used by banks and airlines and even the military.

What these computers had in common was that they all relied on open source software.

A strange fact about modern life is that most of the computers responsible for it are running open source software. That is, software mostly written by unpaid, sometimes even anonymous volunteers. Some crucial open source programs are managed by just a single overworked programmer. And as the world learned last month, these programs can become attractive targets for hackers.

In this case, the hackers had infiltrated a popular open source program called XZ. Slowly, over the course of two years, they transformed XZ into a secret backdoor. And if they hadn't been caught, they could have taken control of large swaths of the internet.

On today's show, we get the story behind the XZ hack and what made it possible. How the hackers took advantage of the strange way we make modern software. And what that tells us about the economics of one of the most important industries in the world. Help support Planet Money and hear our bonus episodes by subscribing to Planet Money+ in Apple Podcasts or at plus.npr.org/planetmoney.

Learn more about sponsor message choices: podcastchoices.com/adchoices

People

Richard Jones, Bruce Perens, Omkar Arasaratnam, Andreas Freund

Companies

Red Hat, Microsoft, Meta

Books

None

Guest Name(s):

None

Content Warnings:

None

Rachel Martin
I'm Rachel Martin. You probably know how interview podcasts with famous people usually go. There's a host, a guest, and a light q and A. But on Wild card, we have ripped up the typical script. It's a new podcast from NPR where I invite actors, artists, and comedians to play a game using a special deck of cards to talk about some of life's biggest questions.

Listen to Wildcard wherever you get your podcasts, only from NPR. This is Planet Money from NPR.

Now that Richard Jones knows how close the entire world came to disaster, he's been looking back for any hints, any clues that he might have missed. For him, the first clue was this message that showed up in his inbox on February 26. So I remember I got this email, and it was not anything unusual. Richard is a senior engineer at Red Hat. He helps make an operating system that is used all over the world.

We're talking Fortune 500 companies, major hospital systems, banks, even the US military. And what's interesting about that operating system is that it is completely open source, meaning it's made out of all these different pieces of software that people are putting out for free. So Richard is often emailing with strangers on the Internet. I don't know who half the people I talk to on the Internet about software are. I don't know who they are in real life.

Richard Jones
I've never met any of them. Instead, we work on reputation. And that email he had gotten, it was from a guy who was new ish on the scene, but who had built up a pretty solid reputation, a guy named Jia Tan. For about a year, Jia had been the volunteer in charge of this very popular software program called XZ, which helps compress data. It's not the fastest, but it is the one that compresses the most.

It's very useful for us. Look, I don't think I'm exaggerating when I say that compression is key for everything, for storing files, for sending stuff over the Internet, everything in the email. Gia sounds pretty enthusiastic, but in a way that a lot of open source volunteers sound enthusiastic. He says, hey, I just made this cool update to XZ. Hope you guys can put it into your operating system.

And this email, I will say it looks very innocent. It's written in this chipper tone. It's got smiley emojis. It has exclamation points which just signals, you know, no threat here. And so Richard goes ahead.

He puts the new updated XE code into a preliminary version of their operating system to test out. But pretty soon, he starts getting these. Bug reports they were quite strange, but not totally unusual. This new version of XE, it seemed to be messing with other parts of the computer, like critical parts of the memory. But, you know, bugs happen in software, right?

Richard Jones
You know, software is full of bugs. So Richard emailed Gia and asked him to, you know, take a look at the problem. He came back within two or three days and said, I'm really sorry. We've just released a new version which fixes this bug for you. So could you upgrade to that which Richard does?

And everything seems fine until about a month ago. That is when someone discovers that this new version of XC, it is not what it seems to be, and Gia Tan is not who he seems to be. I was surprised. I was a bit shocked. I was angry.

Richard Jones
I just didn't expect that somebody would try that. What we now know is that Jia Tan was a hacker, or probably a group of hackers, and they were trying to pull off one of the most audacious cybersecurity attacks in history. Over the course of two years, these hackers had infiltrated one of the most popular programs out there, XZ. And if they hadn't been caught, they would have had a secret backdoor to some of the most important computers in the world. Hello, and welcome to planet money.

I'm Jeff Guo. And I'm Nick Fountain. If you peek under the hood of the Internet, what you'll find is that most of the computers powering it are running free, open source software. But here's a dirty secret. A lot of that software is written by small teams, sometimes teams of only one person, which makes them pretty vulnerable and easy to infiltrate.

Today on the show, the story of the Xe hack, how it took advantage of the strange way we make modern software and what it tells us about the economics of one of the most important industries in the world.

This message comes from NPR sponsor RSM. Change waits for no one. But when it happens, and it always does, be prepared to take charge with RSM's proven advisors, who make it their business to fully understand yours. RSM brings human insights powered by technology so you can leverage the knowledge of future focused minds who look beyond the ordinary RSM experience the power of being understood. Take charge now@rsmus.com.

Spotify. Drake and Kendrick Lamar have been lobbing some serious accusations at each other. You've probably heard the diss tracks and wondered what's just a low blow and what's actually criminal. I'm Brittany Luce, host of it's been a minute from NPR, and I'm getting into what's art and what's worthy of criminal investigation and who those accusations hurt the most. On its been a minute from NPR.

The hack that were talking about today, the XC hack could not have happened if it werent for the weird way that most modern software is made. We have these trillion dollar corporations working side by side with unpaid, sometimes even anonymous volunteers to write the software that powers the Internet. Right. And so before we can get into how the XZ hack went down, were first going to have to understand how modern software got to be this way. For that, we went to one of the founders of the open source software movement, Bruce Perrins.

Bruce Perens
Back when I was starting this, you would have asked what drugs I was using and where you could get some. Bruce says for him, it all started with this epiphany he had about how to write software efficiently. At the time, it was the 1980s, and he was a young programmer at Pixar. He wrote software that helped make movies like Toy Story two. I'm a bigger fan of four, but whatever.

Bruce would keep running into this annoying problem. His programs, they would glitch out, and they would accidentally overwrite other parts of the computer. Yeah, and, you know, these bugs, they would happen all the time. Back then, it was like the wild west of software. So Bruce's solution was to write up a piece of software that would monitor other programs, and it would alert him whenever a program started to overwrite stuff that it shouldn't be writing over.

Bruce Perens
It would stop your program in the instant that happened and let you see the exact instruction that you had wrong. And I called this electric fence because when you touch the fence, it would zap you. Bruce's electric fence proved to be pretty handy, so he shared it with his colleagues at Pixar. But then he thought maybe other people would find it useful, too. And at the time, there were these online bulletin boards where programmers would hang out, and Bruce was a regular.

He says this community had a culture of sharing. Programmers from different companies would show each other new ways of doing things, even provide code for others to copy. So all of the engineers, software engineers at the time, started sharing software together, and we started using it in our work, and no one in management or legal knew that was happening. Bruce went ahead and he posted the code for electric fence onto that bulletin board. And electric fence became incredibly popular.

All of a sudden, people around the world started using it. Someone wrote to me and said, your electric fence program has just saved my job. If you ever go to Ireland, please stay in my home. Oh, my gosh did you go, no. No, I never met this guy.

Bruce Perens
But, you know, I was getting fans. That felt pretty good. But then something even more gratifying started to happen. Some of Bruce's fans, some of these programmers he'd never met, they made their own improvements to electric fence. And they shared these improvements with Bruce.

They sent him their tweaks and upgrades to his code. And I thought, you know, whatever work I've put into this, I just got back and it kept happening. This experience helped Bruce realize two things. First, it illustrated an unexpected benefit of giving away your code, because not only could your code help other people, these total strangers, but now those strangers could help make your code better. And second, Bruce realized that if they all worked together this way, they could write software so much faster and better.

Yeah, Bruce says a lot of programmers in their day to day jobs would spend hours doing essentially the same thing as their counterparts at other tech companies, writing the same basic code to solve the same basic problems. So back then, it would be like building a house, and you'd have to dig up this clay and fire all the bricks. And open source was the idea was we would all chip in on making all the different bricks, and we would give them to each other for free. And now you are not wasting your time on the bricks. You're building the architecture.

This is where Bruce saw the beginnings of a powerful idea. Cause there's already this culture of sharing software in order to be a good citizen or to promote a free society or whatever. But Bruce saw that this could also transform the whole economic model for making software. Bruce believed that if people could get together to produce software in an open and crowd sourced way, you could outcompete even the mightiest corporations. Now, Bruce wasn't the only one who realized this.

By the 1990s, a lot of programmers were attracted to this open source models, but, you know, they weren't taken too seriously. But the open source movement would soon have an opportunity to prove itself. Yeah, you see, by the late nineties, the Internet was really starting to take off. Remember, this was the era when there were so many AOL online CDs floating around, people were using them as coasters. What a time to be alive.

And the big question at that time was, what software would this new Internet run on? And a person who would play a pivotal role in the war over the Internet was Sam Ramji. Software was becoming infrastructure. It was becoming the roads and bridges that we built the emerging Internet out of. In the mid two thousands, Sam worked for one of the biggest software companies in the world.

Microsoft. Here's where we mentioned that both Microsoft and the Gates foundation are funders of NPR. And Microsoft wanted this emerging Internet to be built on top of Microsoft stuff. Microsoft operating systems software that Microsoft owned and controlled. I was in the business development group working in Silicon Valley trying to get startups to adopt more Microsoft software, which was a hard sell, actually.

Yeah, because the open source alternatives, they were becoming popular. Sam started to notice that all the new hot startups, companies like Google and Salesforce, they weren't interested in what he and Microsoft had to offer. No. Most of these startups, they were cobbling together something diY. They were using a free open source operating system called Linux and free open source software that ran on top of Linux.

It was open source on top of open source. On top of open source. You had a stack of software, right? The whole stack just made sense together. And yeah, a lot of the open source software at the time was kind of janky.

There were bugs, there were missing features. But Sam, he noticed that there was also this kind of snowballing effect. The more startups that were building their products on top of that open source software stack, the better that software became. This was this huge emerging economic movement of how software was going to get shipped, licensed, distributed, used, and Microsoft was nowhere to be found. So Sam writes this kind of cheeky memo to his superiors, tells him, look, Microsoft is not going to win the battle over the Internet.

It never will. We've already lost. Open source is the future. If we can't figure out how to work with and use open source software, we're going to go out of business. So your message was, there's no way we're going to beat them.

We have to join them. That's exactly right. And so you send it off and what are you feeling? How do you think they're going to receive it? Trepidation.

This memo eventually makes it all the way to Bill Gates. And to Sam's surprise, the higher ups at Microsoft are like, yeah, you do have a point here. And so they promote him. They put him in charge of open source strategy at Microsoft. They ask him to help turn around the ship.

And this was a huge deal. Microsoft had kind of pioneered the idea that people should pay for software and now it was changing its business model slowly to embrace open source. Over the next decade, it stops seeing Linux as the enemy and it starts making sure that Microsoft software works on Linux, even uses Linux on its own servers. Also, Microsoft starts giving away some of its own software for free, making it open source. Sam says that's extremely common these days.

All the major tech companies do it, like Meta, for example. They started sharing all the tools they use to make interactive websites. Yeah, and to be clear, these companies aren't giving away all their software. Like Meta is not giving away the Facebook algorithm. But what they've realized is that it's more valuable for them to share some of their internal software and have the public suggest fixes or build off of it than it is to keep all of it secret.

Open source is now the default way to make modern software. Bruce's dream of having this library of open source building blocks, these free bricks for anyone to use. That dream came true. Those free bricks are now the foundation for most of the software we use today. But theres also a weakness to this open source model, a weakness that became painfully obvious when the XZ hack went down.

That is, after the break.

Jasmine Morris here from the StoryCorps podcast. Our latest season is called my way, stories of people who found a rhythm all their own and marched to it throughout their lives, consequences and other peoples opinions be damned. You wont believe the courage and audacity in these stories. Hear them on the Storycorps podcast from NPR.

The economy right now is bewildering, impenetrable, inconceivable. Not when you have the indicator of bug eyes in your ears in under ten minutes. Every day, we simplify the complicated news like how does inflation, what the heck is this sPac? Why are trendy little high fiber sodas suddenly dominating store shelves? And more.

Listen to the indicator from Planet money and NPR on the Code Switch podcast. Conversations about race and identity dont begin or end with the news cycle. Thats because we know race and identity impact every person and influence every story. Were getting into all of it with new voices each week on the Codeswitch podcast from NPR.

Rachel Martin
From the campaigns to the conventions, from now through Election Day and beyond, the NPR Politics podcast has you covered. As Joe Biden and Donald Trump Square off again, we bring you the latest news from the trail and dive deep into each candidate's goals for a second term. Listen to the NPR Politics podcast every weekday. Darian woods here. As the US federal debt grows, so too does the interest on it.

And this year it hit a milestone. Interest payments this year will actually be larger than national defense spending for the first time. And that's not a small number. That is one of the largest items in the entire federal budget. That's from our latest bonus episode.

It's my conversation with a long time debt hawk about the potential risks to the economy. And when spending makes sense, you can check that out. Now if you're a planet money listener, if thats you, thanks for your support. If its not, it could be you get bonus content sponsor, free listening and support the work of Planet money. Go to plus dot npr.org dot.

Theres this kind of famous cartoon about how the Internet works. You might have seen it. Its from the webcomic XKCD. Its this drawing of a giant Jenga tower, all these blocks stacked on top of each other. And the whole thing is balancing on this one tiny, skinny little block.

Rachel Martin
I know exactly what you're talking about. And it all rests. The entire Internet relies on this one guy in Nebraska. Umkar Arasaratnam is not that one guy in Nebraska, but he thinks a lot about the Jenga Tower problem. He's the head of the open source Security foundation.

And he says, yeah, they worry about how fragile this whole Internet Jenga Tower is. See, open source software is this huge decentralized community of people building software on top of other software on top of other software. And that is an incredibly efficient way of making software. But it can also lead to these weak spots. Which brings us back to the story of XZ.

In this story, the proverbial guy in Nebraska is, well, not in Nebraska. We actually couldn't confirm where he's from. He wouldn't return our emails, but his website's hosted in Finland, so a lot of people think he's finnish. Anyway, his name is Lasse Kotlin. He's the main creator of Exe Amkar.

Remembers when Lasse first published Exe back in 2009? It was one of these breakthroughs in compression, right? It was one of these things where, oh my God, this literally got two to 300% increase in compression performance overnight. And so everyone started using XC building programs on top of it. Xe became one of the most widely distributed programs in the entire world.

Rachel Martin
There's a good chance it's on your phone, there's a good chance that it's on your tv. It's everywhere. This, this is how the Jenga tower problem starts, how the whole world can come to depend on one random person. Omkar says this is pretty common, that there are a lot of critical software projects that rely on just one person. And the big problem with this is that it is an ongoing job.

Software isn't just a thing you write once and that's that you gotta maintain it. Computers change, operating systems change, new processors are released, new kinds of computers come out. And thus we have to keep our software up to date or it rots. And someone needs to oversee all of these small little updates. It's not the most glamorous work.

Most open source volunteers wanna be contributing to new projects, not looking after old ones. So for many years, the work of maintaining Xe falls to Lasse. Fast forward to 2021. This is when the hacker, or hackers calling themselves Gia Tan, come onto the scene. And here's what we know about how their ingenious plot unfolded.

This Gia Tan character basically appears out of nowhere and soon starts suggesting some improvements to XZ, which is great. This is how open source is supposed to work, right? What makes it so special? Strangers on the Internet helping each other out. Heartwarming.

But a few months later, Lasse starts getting these emails from users of Xe. They're complaining that lase been falling behind on maintaining Xe. One of them's kind of nasty. It's saying how sad it is that Lasse clearly does not care about this project anymore. Hey, this has been delinquent for a long time.

Rachel Martin
How come nobody's updated this? When are you going to get to it? That kind of thing? That's pretty rude. I'm sorry.

Like, this guy's doing it for, well. You know, this is the, I guess this is one of the failure modes of how society has consumed open source. The overhead of having to deal with this stuff can become overwhelming. Lasse tells these people, you know, I'm sorry, the work is going slow. I'm dealing with some personal stuff right now.

But his critics are still not satisfied. Someone suggests, why doesn't he just step down and let someone else manage this thing? And pretty soon thats what Lasse does. He decides to pass the baton on to that new volunteer, Jia Tan. Now, Jia is going to be the one holding up that Jenga Tower instead of Lasse.

Omkar says, what we know now is that Jia Tan was probably an invented personality. But also these people harassing Lasse, they too seem to be invented personalities, people who were created just to convince Lasse to pass that baton to Gia. It was literally a social engineering attack. Somebody basically running a long con and tricking Lassay into doing things and giving permission that they shouldn't have. Gia takes over.

And over the course of the next few years, Gia starts to make all these little changes to Xie, seemingly innocuous changes that start to turn XZ into a trojan horse. You see a lot of programs depend on XE, including a very important program called OpenSSA. It's basically the garage door opener for the Internet, it lets you remote control other computers. Pretty much every web server is running it. It is literally the thing that controls access to every server on the Internet.

Rachel Martin
It is really important. This garage door opener program is a really well guarded piece of software. Everybody has their eye on it. What Ga tan, or what the hacker group behind the identity known as Ga Tan had figured out was that if they could secretly sabotage XZ, they could sabotage this garage door opener and give themselves access to basically every important computer on the Internet. This was incredibly well orchestrated.

I think somebody should make a movie about this. I mean, I'd definitely watch it. I'd watch it in IMax. Earlier this year, Gia starts pressuring the major open source operating systems to use their new sabotaged version of Exe. That's when they send emails to people like Richard from the top, and the compromised XZ starts slowly spreading across the Internet.

Now, the way this hack was eventually discovered is kind of by accident. It was discovered by this programmer at Microsoft named Andreas Freund, who works on open source software. Actually, a couple months ago, Andreas noted that the garage door opener software was acting kind of slow, and he started. Picking it apart, and he pulled that thread, and he eventually unpacked all the stuff we know. Now Andres sends out an email about this.

He's like, hey, guys, I think one of the most important pieces of software in the world has been compromised. And also, I'm pretty sure this is exactly how they did it. When Omkar sees this email, he almost falls out of his chair. My first reaction was, oh, my God. How many people have downloaded this?

Luckily, the sabotaged XZ was caught before it got widespread distribution and mostly only got onto computers running experimental or beta software. Can you run me through, like, what the nightmare scenario would have been if Andreas hadn't caught this nightmare scenario is. It gets broad distribution. Whoever GTN is quietly logs into computers all over the Internet, stealing money, your personal information, I mean, anything. Stealing your email.

Rachel Martin
It could have been anything. Umgar says it was a pretty shockingly close call, and it has started to make people reconsider the entire economic model of open source. The open source movement succeeded beyond anybody's wildest dreams. It started with these programmers who were writing code in their free time because they thought it was fun or they wanted to make something cool, or they wanted to make the world a better place. But over the last three decades, all those volunteers have built this efficient, decentralized, maybe even beautiful system of writing software, software that became the foundation for the Internet.

Yeah, but out of this efficient and decentralized and beautiful system. You also get the Jenga tower problem, where one person can write a program thats so good it changes the world and it leads to the whole world depending on that one person. Omkar says the solution is not that open source software goes away, but we have to reconsider how we treat the open source software community. He says open source has become this incredibly valuable public good. It's become like the pipes and sewers of the Internet.

And like any public good, there aren't really strong incentives for people to help maintain them. Open source folks are all incentivized to work on the new shiny thing, right? To build skyscrapers. In the meantime, the less interesting projects that we're all relying on, the proverbial sewer pipes, nobody's taking care of them. And when the sewage backs up, we're all in trouble.

How many vulnerable programs like XZ are there? Angkor says there could be a lot. He and his colleagues are working on this giant census to try and identify all the single little Jenga blocks holding up the Internet. He says they expect to have new results later this year.

On our next episode, layoffs. They happen all the time. Theyre a business reality. But of course they can be really destabilizing. Honestly, I felt like I was being.

Swallowed by the sinking hole. Like when this person lost his job, he and his husband had a lot of questions, especially for the HR rep who handled the layoff, like, do you. Get training on how to be human in these conversations? Those questions and more on our next episode. This episode was produced by Emma Peasley and engineered by Cena Lofredo.

It was edited by Jess Zhang and fact checked by Sierra Juarez. Alex Goldmark is our executive producer. I'm Jeff Glow. And I'm Nick Fountain. This is NPR.

Thank you for listening.

The Bullseye podcast is, according to one journalist, the quote, kind of show people listen to in a more perfect world. So make your world more perfect. Every week, Bullseye puts the pop in culture, interviewing brilliant authors, musicians, actors, and novelists. To keep you on your pop culture target, listen to the Bullseye podcast only from NPR and maximum fun. What does it sound like to record an album inside a jail?

On the documentary podcast track change, you'll hear four men make music inside Richmond City jail and hear how they're trying to break free from a cycle of addiction and incarceration. Been so long since I've been free. Listen to track change from narrative lead and VPN, part of the NPR network. You care about what's happening in the world. Let State of the world from NPR keep you informed.

Rachel Martin
Each day we transport you to a different point on the globe and introduce you to the people living world events. We don't just tell you world news, we take you there and you can make this journey while you're doing the dishes or driving your car. State of the World Podcast from NPR. Vital international stories every day.