Chip shots: breaking Nvidia's AI grip

This episode delves into the dominance of Nvidia in the AI chip market and explores emerging technologies and companies aiming to challenge this supremacy.

1. Nvidia's dominance is rooted in its GPUs' adaptability from graphics to AI processing.
2. Companies like Cerebras are designing chips specifically for AI, with potentially superior performance.
3. The shift from general-purpose GPUs to dedicated AI chips could significantly impact AI development efficiency.
4. Market newcomers face substantial challenges in displacing established players like Nvidia, primarily due to software compatibility issues.
5. The potential market value for breakthroughs in AI chip technology is estimated to be in the trillions.

1. The AI Universe

Nvidia's control over the AI chip market mirrors the monopolistic control seen in fictional universes. The episode opens with a discussion on how GPUs, originally intended for gaming, became pivotal in AI. Jason Palmer: "He who controls the GPUs controls the universe."

2. Innovations and Challenges

Exploring the technical reasons GPUs are suitable for AI and how startups are innovating to create more specialized AI chips. Tom Standage: "If you're building an AI chip, try to work out how you can more efficiently connect all of these different cores together."

3. Future of AI Chips

Discussion on the future landscape of AI chip technology, with insights from various startups and big tech firms venturing into AI-specific chips. Tom Standage: "The bounty available here for any company that can break the stranglehold that Nvidia has on this market is worth at least a trillion dollars."

1. Understand AI Needs: Assess if your current AI solutions could benefit from more specialized AI chips.
2. Stay Informed: Keep up with technological advances to ensure your AI infrastructure remains competitive.
3. Evaluate Suppliers: Consider both established and emerging chip manufacturers for AI projects.
4. Consider Energy Efficiency: New AI chips could offer significant energy savings, a crucial factor for large-scale AI operations.
5. Risk Assessment: Weigh the risks of adopting newer technologies against the potential performance benefits.

When it comes to the chips used in artificial intelligence, one firm has the market locked up. We look at the rivals minded to steal Nvidia’s crown. The death toll from the war in Gaza has been disputed since the start; we cut through the numbers to find a reliable estimate (10:19). And our correspondent examines the great rematches of fiction (16:07).

People

Jason Palmer, Tom Standage

Companies

Nvidia, Cerebras, Grok, Google, Amazon, Meta, OpenAI

Books

None

Guest Name(s):

Tom Standage

Content Warnings:

None

Yahoo Finance
When it comes to your finances, you think you've done it all. You've saved, you've researched, and you've invested all that you can. Now it's time to take those investments to the next level by using the brand behind every great investor. Yahoo. Finance as America's number one finance destination, Yahoo Finance has everything you need, whether you're a seasoned trader or just dipping your toes into the market.

Join the millions of investors who trust Yahoo Finance to guide them on their financial journey. For comprehensive financial news and analysis, visit Yahoo Finance.com comma. The number one financial destination yahoofinance.com dot.

Jason Palmer
The Economist hello and welcome to the intelligence from the Economist. I'm your host, Jason Palmer. Every weekday, we provide a fresh perspective on the events shaping your world.

It's all but impossible to know with precision just how many lives have been cut short by the war in Gaza, and many have quibbled with the death toll published by the Hamas run ministry of health. We look at how reliable the numbers are and have been, and a repeated face off between american presidential candidates used to be a more common thing. This year's is the first since 1956. Our correspondent mines the great rematches of fiction to set us up for a coming one of trumpian proportions.

But first.

Tom Standage
He who controls the spice controls the universe. That is one of the more famous lines from doom, a science fiction epic. Know, then, that it is the year 10,191. In this time, the most precious substance in the universe is the spice melange. In it, the spice is a prized drug that extends life and permits faster than light travel.

Jason Palmer
And whoever controls the supply, well, they get filthy rich and they control that fictional universe.

Here in the real world, among the people who think about artificial intelligence, that line often gets refrained, framed. He who controls the GPU's controls the universe. GPU's, simply put, are the chips at the heart of every AI breakthrough you've been hearing about. And there's one company that has the market locked up, Nvidia. Yesterday it reported quarterly revenue numbers.

They're up 262% year on year. So for now, anyway, Nvidia controls the AI universe. The weird thing about Nvidia's dominance of the market for AI chips is that the chips that it's selling were originally made to do something else. They're GPU's, and that stands for graphics processor units. They're made to do video game graphics.

Tom Standage is a deputy editor of. The Economist, and it turns out that the chips that were made to do graphics could be repurposed to do AI and therefore Nvidia suddenly found itself sitting on a gold mine. So what a lot of people have said is, well, hang on a minute, if you were going to make a chip to do AI, you wouldn't start from here. What would a from scratch chip design look like if you were optimizing it for AI? And so there's a bunch of companies trying to do that and they think we'll be using special AI chips in the future.

But how did GPU's end up as the go to chip for this in the first place? Well, the reason that graphics chips turn out to be good for AI is that both graphics and AI involve doing lots and lots, thousands, in fact, of copies of the same process in parallel. And that's because when you're drawing a scene, you've got thousands of elements in the scene and you've got to process them all and figure out the lighting and all of this kind of thing. And it turns out the most efficient way to do that is to have thousands of what are called cores inside the chip. So a modern GPU will have more than 10,000 of these little cores.

Tom Standage
So they're not particularly powerful, but they're really fast and there are lots of them. And it turns out that when you're doing the maths that underpins large AI models, you're essentially multiplying lots of matrices together. That's what you're doing. And so if you've got little processors, little cores that can do that in huge quantities and very, very quickly, then you can do AI. And that's essentially what these graphics chips are doing.

So if you're an AI chip company, you're essentially looking at the GPU and saying, okay, well, that's pretty good, but what are the problems with GPU's? And how can we make a dedicated AI chip that's better? So it's less a question of computing and more one of connecting. So how do you make it more efficient? One of the ways you could do that is that GPU's are connected together in clusters quite often, and the memory that they use is quite fiddly and requires very special interconnects.

And the problem is that shuffling all the data between the GPU's and in and out of the memory is not terribly efficient. So one of the CEO's that I spoke to, who's working on one of these AI chips, likened it to a situation in a supermarket just before the holidays where there's a traffic jam in the car park, there's queues to get a trolley. There's absolute chaos in all the aisles, and no one can move. And so essentially everything is blocked. People are moving and they're buying things, but it could be much more efficient.

And that's sort of what's happening with the data that's flowing between GPU's and memory chips. It turns out that GPU's, when they're training big AI models, the cores inside the GPU are very often idle half the time because they're waiting for data to be delivered. So one of the things you could do if you're building an AI chip, is try to work out how can you more efficiently connect all of these different cores together so that they're not kept waiting. Okay, so give us the recipe, then. How do you make a dedicated AI chip?

So one of the approaches that is being taken is by a company called Cerebrus. And essentially what it's doing is it's taking all of the cores that you'd find inside a GPU chip. It's taking about as many as you'd find in about 100 of those chips. And it's putting them all onto one enormous chip. And when I say enormous, it's literally the biggest chip that's ever been made.

These chips are the size of a dinner plate. They're absolutely enormous. They're 50 times bigger than any other chip. And that means you can have 900,000 cores on one of these chips, and lots and lots of memory. What that means is the connection between the cores is really fast and between the cores and the memory, because they're literally all part of the same chip.

So that means you get really, really fast performance. And the company says it also reduces energy consumption. Okay, so to come back to your supermarket analogy, what cerebras is doing here is giving you thousands of checkout cues so you don't get back into the aisles. What other kinds of ideas are out there? Well, another approach is taken by a company called Grok.

And that's Grok with a queue, not Grok with a k, which is Elon Musk's AI startup. And what Grok is doing is it's using lots of small chips connected together, but they're special sources in the way that they're connected together. So they've got these chips that have memory on them, and each chip acts as a processing unit, but it also acts as a router that talks to the chips next to it. And they've got a clever layer of software that gets rid of the variation in latency between sending packages between different chips. So it's that unpredictability that's the problem.

And what that means is that a whole bunch of its chips connected together can act like a giant chip working in lockstep. And that means that programming it is much easier because everything's much more predictable, and that leads to great efficiency. It's quite a complicated thing they're doing. But GPT four has been running at 20 tokens per second for the whole of this year, and the latest version of it runs at 80 tokens per second. But Grok can run a similarly big model ten times faster than that.

So that's pretty impressive, too. Okay, so this sounds like not computing, not connecting, but more like coordinating. Yeah, it's sort of like there's a conductor in the middle of the supermarket telling everyone to move and predicting when they're going to get to the fish isle or the frozen goods or whatever. So it allows you to sort of coordinate the movement in a very organized way, which means that you're not holding up the cause, they're not wasting their time waiting for things. So there's a bunch of startups doing this.

I think we should also mention that big companies are also trying to get into this market. The most successful is Google, which does already make its own AI chips called tpus, tensor processing units, and it operates them in its cloud, so you have to rent capacity on them. Amazon and Meta are also. They've got their own AI chips that they run in their clouds. OpenAI, the maker of chat, GPT, is making noises about making its own chips as well.

There are companies big and small that are trying to solve this problem. And one analyst said to me that the bounty available here for any company that can break the stranglehold that Nvidia has on this market is worth at least a trillion dollars. Well, as you said at the start, this is a reimagining, not just a copying what Nvidia does and trying to do it a little bit better, is that from the ground up, approach the right lever to pull to get that trillion dollar payout? Well, there are people who are just trying to copy Nvidia. So AMD and intel, who are two sort of incumbent chip makers, they also make GPU chips essentially like Nvidia's, and you can use theirs instead.

And some people do if you want to, but they just don't have the market share and the mind share. So I think what these other companies are trying to do is they're trying to really leapfrog that whole approach, and not just out compete Nvidia on its own territory, but really start from scratch and think of a completely new way of doing things. And some of them have got some really quite radical ideas. The problem, I think, is that Nvidia has a tight lock on the market because the hard part about all of this is getting your software to work on these new chips so it's easier. Just stick with Nvidia and use its system and its chips.

And that means that trying to break its stranglehold on the market is really quite difficult. So, as so often the case, inertia is the bigger force at work here. I mean, is any of these startups onto something that might break that stranglehold in the long run? Well, I think the optimistic case is that one of them could find that as AI models improve, their chip is particularly well suited to running whatever models look like in a couple of years time. So far, none of these startups has made even a small dent in Nvidia's dominant position.

It really is the 800 pound gorilla here. That doesn't mean, however, that one of them won't make some breakthrough in the future. And there are plenty of people hoping that one of them will. Thanks very much for joining us, Tom. Thank you, Jason.

ServiceNow
Here's the truth about AI. AI is only as powerful as the platform it's built into. ServiceNow puts AI to work for people across your business, removing friction and frustration for your employees, supercharging productivity for your developers, providing intelligent tools for your service agents to make customers happier, all built into a single platform you can use right now. That's why the world works with Servicenow. Visit servicenow.com aiforpeople.

Jason Palmer
From the very beginning of Israel's war in Gaza, the death toll has been disputed. The numbers used to come only from Gaza's ministry of health, or MOH, which is controlled by Hamas. A week into the conflict, critics accused the MoH of exaggerating the number of fatalities after an explosion at al Ahli hospital.

Since then, there's been relentless scrutiny of the MoH's figures, which are updated several times a week and repeated by the UN and media organizations, including us. Statistically impossible. That's how three australian academics have described the gazan death count. Adding to the confusion, on May 8, the UN's website appeared to revise down its count of the women and children who had been killed. In addition, the UN faces controversy regarding.

Their Gaza death toll as some what changed? And have any of these numbers at any point been reliable? One of our data journalists Ainsley Johnston. Aimed to find out at the beginning of the war there's evidence that the death toll from Gaza's ministry of health or the MoH was relatively accurate at that point. It was based entirely on deaths that were registered at hospitals and morgues in Gaza in previous conflicts.

Ainsley Johnston
The figures from the Ministry of Health have had a pretty good match with independent counts of death tolls. But as the war has dragged on, the quality of the MoH death count has deteriorated. Well, lets wind back to when things were clearer. How do we know that the death toll at the beginning of the conflict was accurate? So on October 26, the MoH released the names, ages and ids of every person they claimed had died in the war in Gaza.

At that point the death toll stood at around 7000, around 68% of whom were women, children or elderly people. Two academic articles, both published in the medical journal the Lancet late last year, analyzed this list of names, ids and ages and both concluded that the list seemed to be legitimate. But you say that as time has gone on, the numbers are becoming a little less believable. Why is that? Yeah, exactly.

So at this point, only three of the eight hospitals that normally register deaths through the system are providing data. In around mid November, the MoH began to supplement the death count from hospitals with deaths recorded through media reports. Media reports are a common way for governments and other organisations to try and estimate deaths and wars, but they can be fallible. So media reports might focus on recording the deaths of innocents rather than militants. Some deaths can be missed and others can easily be double or even triple counted.

As time has worn on, commentators, particularly pro israeli ones, have been quick to point out that there are inconsistencies in the MoH figures. There are times when the supposed male death toll drops from one day to the next, and there are odd patterns in the numbers of men, women and children who have died each day. And this leads some people to believe that these numbers are just bogus. So how then to get hold of numbers that everyone can agree are not bogus? So there are some credible figures still, the MoH has continued to publish these named lists of deaths at sporadic intervals.

These lists only include deaths registered through the hospital system, and more recently through this third system, which is a Google forum online where people can register the deaths of relatives and friends. The UN, the who and Human Rights Watch all think that these lists are pretty trustworthy. And the switch on the UN website reflects a change to using the figures from these lists. There was even an analysis conducted by the Israeli Defence Forces, which was seen by the economists. They looked at the list published on the 6 January and concluded that the vast majority so 83% of the around 14,000 people who were on this list were real people whose names and ids matched official records.

Around 1400 of those people on the list were verified Hamas militants. There have also been more recent updates of this named list that also appear to be relatively error free. So our own analysis of the newest list up to April 30, we find that around 84% of the ids were valid. And so with that comparatively trustworthy list, then what conclusions can we draw about the number of people who have actually died in Gaza? So the total death toll that the MoH has been putting out is a lot higher than the numbers that we see on this named list.

But if we rely purely on this list, we can say with moderate confidence that as of the end of April, at least 24,600 Palestinians had died during the conflict, and of those, at least 13,800 are women, children or old people. This only represents around 70% of the total number of deaths that the MoH think have happened. Even Israel have indicated that they expect the true death toll to be a lot higher than the MoH named list would imply. In March, Benjamin Netanyahu, Israel's prime minister, claimed that up to 32,500 people may have been killed in Gaza and that around 60% of them could have been civilians. Whatever the true figures, the loss of life so far has been immense.

Jason Palmer
Thanks very much for your time. AINSLEY thank you, Jason.

Andrew Miller
Once upon a time, rematches were common in american elections. Andrew Miller is a special correspondent at the Economist. But they've become rarer as politics has become an unforgiving one. Strike and you're out game. Donald Trump is an exception to this rule, as he is to many others.

Tom Standage
This will not be my campaign. This will be our campaign altogether. His rematch with Joe Biden this November will be the first in american elections since the 1950s, when Eisenhower defeated Stevenson twice. But while rematches have become less common in national politics, they are staples of films and literature. And since we're all going to be living with a rematch for most of this year, I wondered what the great rematches of fiction could reveal about the stakes and motives and drama when old rivals clash again.

Andrew Miller
One way to think about rematches is to divide them into two categories. In the first, the outcome is unchanged, and whoever was stronger or faster or cleverer in the original clash remains so in the second, think, for example, of probably the most ill fated rematch in literature between Captain Ahab and Moby Dick. In their first encounter, Ahab loses a leg to the white Whale, and in the second, he loses life in his ship. And the moral of that story is really to know when you're beaten, and that trying to get even, as Ahab does, can take a lot of other people down with you.

In the second category of rematches, the losers regroup and lick their wounds, learn something, maybe about themselves, and come back and get their revenge. For example, in the Star wars trilogy, Darth Vader fights Luke Skywalker twice. In the Empire strikes Back, released in 1980, Darth Vader cuts off Luke Skywalker's hand, and worse from Luke's point of view, he reveals, I am your father. They meet again in Return of the Jedi, released in 1982. Your thoughts betray you, father.

Jason Palmer
I feel the good in you, the conflict. There is no conflict. Luke swears he wont fight his father, but then he does and wins.

Andrew Miller
What may be modern cinemas best known rematch wasnt really supposed to happen. In the 1976 film Rocky, the heavyweight champion Apollo Creed beats the title character Rocky Balboa on points after an epic slugfest, and both the fighters agree not to take each other on again.

But then the film made a fortune and they changed their minds. Fast forward to Rocky two, released in 1979, and Creed pushes his luck, as incumbents sometimes do, and challenges Rocky to another bout. Man, I won, but I didn't beat him. At the climax of this fight, they're both on the canvas, but Rocky manages to beat the count and staggers to glory.

Now, not all of fiction's great rematches involve violence. Some are what you might call tussles of love. Take, for example, pride and prejudice. When Mister Darcy first proposes to Elizabeth Bennett, he makes some rookie errors. For example, he insults her entire family.

Colin Firth
You must allow me to tell you how ardently I admire and love you.

In declaring myself thus, I am fully aware that I will be going expressly against the wishes of my family, my friends, and, I hardly need add, my own better judgment. Darcy was memorably portrayed by Colin Firth in the BBC adaptation of 1995. And by the time of his second proposal to Elizabeth, he's mastered what Jane Austen calls his abominable pride, and he does a better job of it. As a child, I was given good principles, but was left to follow them in pride and conceit. Such I might still have been.

But for you.

Andrew Miller
Another way to think about rematches beyond just the question of winners and losers is as a yardstick of change. Will the heroes outdo their former selves, or will they be undone by time?

As for what these fictional rematches and archetypes can tell us about the election of 2024, well, sadly, the parallels are imprecise. Biden v. Trump, too, for example, is not quite as unique universally and eagerly anticipated as was rocky two. Donald Trump doesn't really seem to have experienced the kind of moral education that Mister Darcy does in pride and prejudice, and neither Biden nor Trump is exactly a skywalker in terms of their asian generation. In any case, in truth, elections are less a one on one standoff than an accumulation of micro dramas in which tens of millions of american voters will decide whether to stick or twist.

I think that's really the key to the theme of rematches in art, because it's not just polling day. Life, for most of us is mostly a rematch. When you come to think about it. It's a series of run ins with the same people, the same relatives, the same bosses, the same issues, and the same you. In rematches, we glimpse the possibility that things might one day turn out differently.

Or we glean the wisdom to accept that they probably won't.

Jason Palmer
That's all for this episode of the intelligence. But have you heard our subscriber only Saturday edition yet? Listen up, youve got one more free episode coming of the weekend intelligence, our special end of the week episode that dives into poignant, compelling, and sometimes very personal stories from our correspondents. Last Saturdays was a dispatch from Mumbai, a thoughtful and sometimes frustrated love letter from my colleague Leo Marani to his home city, one that makes you think about who exactly, any city is really for. Oh, and we hear why he insists on calling it Bombay.

So mark your calendar for the last free episode, and then, like very next thing, get half off a subscription to Economist podcasts plus, so you can get that Saturday kickback and all of our weekly shows all the time. Plus deep dive series like the one starting in a couple of weeks about Narendra Modi. It's a couple of bucks a month, but hurry only for another week. Search for Economist podcasts and get your subscribing thumb ready.

ServiceNow
Here's the truth about AI. AI is only as powerful as the platform it's built into. ServiceNow puts AI to work for people across your business, removing friction and frustration for your employees, supercharging productivity for your developers, providing intelligent tools for your service agents to make customers happier, all built into a single platform you can use right now. That's why the world works. With ServiceNow, visit servicenow.com aiforpeople.