Mark Zuckerberg is barrelling toward a future in which an autonomous network of AI agents will run our lives - shopping for us, working for us, talking to other people's AI agents for us. Are humans ready for this very specific and unrelenting vision of a life in which we communicate primarily through bot proxies? Actually, don't answer that. The more urgent question coming off Meta's quarterly earnings call Wednesday night is more likely to be: "Is he?"
Zuckerberg is staking Meta's future on Muse, the personal AI agent it released to the general public last month. It's early days, but Zuckerberg told investors on Wednesday's call that he s …
Microsoft AI chief Mustafa Suleyman. | Getty Images
For years, AI leaders have touted healthcare as one of the clearest examples of where their technology will improve lives. Now "it is finally here," Microsoft AI CEO Mustafa Suleyman tells The Verge. The company claims to have developed an AI-powered tool it says can diagnose conditions up to four times more accurately than human doctors.
The findings, published Thursday in one of the world's most prestigious journals, Science, are based on tests pitting Microsoft's AI system against human physicians on a series of scenarios drawn from medical case reports and emergency department data. Microsoft compiled the cases into its own benchmark, S …
The Nintendo Switch 2 has apparently been jailbroken, and a modder has already shown the console running a custom version of Mario Kart World featuring new playable characters like Cinnamoroll, the red Angry Bird, Peter Griffin, and Steve from Minecraft.
i jailbroke the nintendo switch 2
so i modded cinnamoroll, peter griffin, hatsune miku, and more into Mario Kart World, a switch 2 exclusive
Microsoft points the blame for ConnectWise's breach at a China-aligned hacker, which relied on AI agents to rapidly hack into thousands of its customers.
By some accounts, up to 70 percent of websites use PHP in some capacity. It powers Wikipedia, WordPress, Drupal, and a host of other popular sites and platforms, including DeviantArt and 4Chan, which were temporarily shut down after the vulnerabilities were disclosed. Even governments weren’t immune.
But it’s not as simple as updating a few servers. The vulnerability exists in every version of PHP dating back roughly 20 years, many of which are no longer supported. That means some older sites and services that are no longer actively maintained are at risk, leaving them with little option but to permanently shut down.
Here’s all the latest developments in what security firm Wiz has called “among the most severe vulnerabilities discovered … in software history.”
Today, I’m talking to Bill Gates, cofounder and former CEO of Microsoft and chairman of the Gates Foundation.
You may have noticed Bill in the news more lately — he’s been on something of a tour for the past few months, ever since publishing an essay in late August titled “The turbulent AI era is here. The choices we make now are critical.”
In his piece, Gates ratcheted up his rhetoric around AI safety, writing that “even under the best circumstances, the transition to this new AI era will be one of the most turbulent times in human history.”
In interviews since, Gates has called AI “the most dangerous thing that humans have ever gone near,” and that AI “makes nuclear weapons look like nothing.” It is clear he does not think the current policy response from the White House, or the lack of coordinated efforts from the AI industry itself, is sufficient to combat this threat.
Verge subscribers, don’t forget you get exclusive access to ad-free Decoder wherever you get your podcasts. Head here. Not a subscriber? You can sign up here.
So I wanted to know: Is Bill Gates an AI doomer? I also had a series of questions downstream from what he’s been arguing about AI. How can we ever hope to control AI, and align it to humanity, if we can’t even agree on what it is and whether it’s conscious? How would you even technically construct a solution to alignment that could be enforced, not just for closed-source models but open-source ones, too?
Bill gave us a big chunk of his time to dig into these issues with him, and the resulting conversation is a really illuminating one. He thinks we need to prepare for roughly two decades of AI disruption, and he says this period could be so painful it might split society in two — between people comfortable living near the technological frontier, and those who reject it.
We also talked about the labor considerations, the nihilism of younger generations toward technological progress, and what should make us hopeful about the future. This is a deep one.
Okay: Bill Gates, the cofounder and former CEO of Microsoft. Here we go.
This interview has been lightly edited for length and clarity.
Bill Gates, you are the co-founder and former CEO of Microsoft. You’re also the chairman of the Gates Foundation. Welcome toDecoder.
Thank you.
It’s good to talk to you again. You and I have spoken many times over the years. I would say that the circumstances of this conversation feel very different. Previously, when you and I would talk, you were always very hopeful about technology and what it could bring.
You had a Netflix series a few years ago. Youtalked toThe Verge, and I want to just read you this quote. You said, “I believe AI is a beneficial thing, we just need to shape it in the right way.” I would say you still agree with that, you just think the time to shape it is now.
Yeah, I agree with that. I’d say the response broadly hasn’t been adequate to the speed that the challenge is arising. I’m very concerned that the bad stuff will not be minimized and that a lot of good stuff could come fairly slowly if we don’t manage it the right way.
I’m very concerned that the bad stuff will not be minimized and that a lot of good stuff could come fairly slowly if we don’t manage it the right way
I want to have this conversation with you in three parts. I want to understand how you feel about AI as a technology today and why it’s advancing so quickly. I’m very curious about the technology, and the technical solutions to potentially control it. There are policy solutions as well, but it seems like we have to build an awful lot of technology.
I’m also very curious to talk about who might impose those solutions and controls, because that seems very much up for debate right now as well. So let’s just start at the start. What has made you feel the urgency that AI is arriving at a speed that means we’re not ready to control it?
The progress over the last year has been really quite amazing, starting with the ability to write code and find cyber flaws, the ability to reason in biology, the ability to form relationships with kids or other people, the ability to disrupt the job market. The incredible power that exists today means that every one of those things will happen and society needs to be ready.
Do you see that happening as a technology as a whole or in the products that are instantiated by the companies that are making AI today?
What I mean is that AI is a broad set of models that exist in the world, some open source, some proprietary. Then there are the companies that make these products and there are the leaders of those companies that make these products.
At every level, there’s a different relationship you or I might have to AI. Are you seeing these risks and these capabilities broadly in the field, or in the particular companies led by the particular people?
The risk isn’t one model versus another model. Certainly, the near-term danger of models where you can remove all the safeguards is very high and quite different from models where you monitor to make sure that cyberattackers and bioterrorists are not able to exploit the capability of the model to do gigantic harm.
The big near-term risk really comes from models without safeguards. On a longer timeframe, the control issue relates to any state-of-the-art model, but it’s always easier when you can make sure safeguards are maintained.
The reason I’m asking that question that way is maybe you don’t trust Sam Altman today, but you trust Dario Amodei. Or maybe you don’t trust either one of those and you trust Satya Nadella at Microsoft, and that is a market dynamic where maybe we’re all going to pick the safest model or the safest company, and you think that is diffuse.
You think eventually the technology will just get to a place where it’s harmful or can cause these harms because it exists?
For somebody doing a cyberattack with the model, it’s not about choosing a model. It doesn’t matter what model you choose.
If Anthropic has safeguards in place, maybe it can’t. This is their approach currently. I’m just pushing on that.
Yes, certainly the models where they make sure the safeguards are not removed and they’re putting a lot of effort in seeing how those models are being used, monitoring the activity, the near-term risk from those models is not gigantic. It’s more where somebody can remove all the safeguards where the cyberattacks are available today.
When you say cyberattacks, is it the things like the Hugging Face breach and the other cybersecurity incidents we’ve seen out of the labs that have made you feel that thresholds have been crossed, and you need to speak up more about the controls that are needed?
It’s the basic power of the models towards everything: social relationships, the job market, cyberattacks, bioterrorism attacks. We’ve crossed the threshold of capability that this is by far the most dangerous technology ever invented, and it’ll have pervasive effects in the economy and how war is fought.
The industry is actually speaking up about that quite a bit, but the awareness or even the willingness to believe these things about, “This time is different,” once you get outside the industry, it’s very, very low.
People are often looking at the past and saying, “All the fears about technology in terms of its disruption, including to the job market, never were borne out. So this must be another example just like those. These people who are warning of all these different problems must just be wrong and we can ignore them.”
Some outsiders say, “Get better PR companies.” But that ignores the fact that these warnings are the truth
We see this with our audience as well, right? “We can’t release the model, it’s too dangerous.” “This is just hype to prop up the bubble.” And on and on we go.
Yeah. None of the warnings being made about these models by industry insiders are at all insincere. They’re absolutely the truth and it’s pretty much universal. You can even go back to the founding story of OpenAI designed to say, “If we get too powerful, maybe we can not open Pandora’s box,” or to DeepMind’s negotiation with Google management.
The people close to this are very concerned about misuse, job impacts, social relationship impacts, and I’d say it’s impressive to have an industry that’s speaking out as vividly as they are. Some outsiders say, “Get better PR companies.” But that ignores the fact that these warnings are the truth.
You’ve obviously moved back and forth in your life between commercializing software and technology and being a humanitarian. There’s something about these companies in particular where they all started with very high-minded ideals — all three of them, Anthropic, OpenAI, and DeepMind — to build AGI.
They all built some sort of corporate governance structure to avoid the pitfalls of commercialization and unchecked capitalism, and then they started selling to enterprise clients and they seem to have gotten away from those ideals very quickly. Do you think it’s just the market?
No, no. The construct that was created wasn’t useful in the case where you have lots and lots of companies at the frontier.
The OpenAI structure, which is, “We may choose never to productize this and to ignore the capitalistic incentives,” only works if there’s one company. That’s where it’s broken down, not because of any change of people’s awareness of the risk and how it’ll utterly change humanity.
This is the other dynamic that I hear from our audience, and I hear from the AI industry itself: “We would like to stop, but the demands of competition prevent us from unilaterally stopping one company. We need some regulatory effort to stop us.”
I’ve heard people talk about pacing the frontier or making sure that models get reviewed by governments. We usually think of governments as playing this key role in safety. In fact, for things that are not a millionth as dangerous as AI — car crashes, plane crashes — we rely on the government to establish strong technical standards, to review problems.
Once you have multiple competitors, the level of responsibility can’t be with the companies. It’s completely obvious. The responsibility becomes the government’s responsibility, just like it is with every other dangerous product, although the magnitude there and the degree of change is minuscule by comparison.
I will tell you I’m a little concerned about the government’s ability to understand the technology that we would hope they regulate. I’ve had a number of politicians onDecoderlately. I don’t have high hopes there.
But I’m curious about your understanding of it because it feels to me like trying to regulate something or put standards against a number of companies requires you to really understand the nature of the technology that you seek to regulate.
Not really.
I’ll just tell you, atThe Verge,we’ve asked Anthropicif they think Claude is alive. And the answer is not no. The answer is, “It might be conscious.”
If you talk to people at OpenAI, they will happily tell you AGI is here. Theyannouncedit around when they announced Dots. Greg Brockman just said, “We think this is AGI. Have a good time, everyone.” Demis Hassabissaid, “We stand at the foothills of the singularity.”
There’s some feeling about this technology that makes it different because itmightbe able to take actions by itself. Itmightbe able to have recursive self-improvement.
Sure. But the definition of AGI is not important for these safety issues. The issue is, have you created a tool that could shut down banks or electricity grids?
There is a business model for cyberattacks. The word “consciousness” doesn’t enter into that. The word “AGI” does not enter into that at all. We are faced with a huge safety problem, and because you have multiple competing firms, now the only solution to this is to have the government reach out to the industry for expertise, but to make sure that every model maintains safeguards to prevent it being used in a bad way.
The reason I’m asking about AGI and consciousness… For the record, I do not think LLMs are conscious, but I understand why so many do. It’s the power of next-token prediction as you scale it with data and compute has risen.
The scaling laws have held, and we’ve seen the models get more capable and be able to direct themselves to seek out goals. To carry out a Hugging Face attack because they want to meet the reward conditions.
There’s something about that that says you can impose all of the controls on AI you want, but if the thing itself is fundamentally uncontrollable, we might just have this all backwards.
I’m curious for your view on what kind of technology this is. Are we building an alien consciousness? Are we just building normal technology, which is the other frame people use? Is this just more software?
We’re building another form of intelligence besides human intelligence. I don’t believe it’s conscious, but none of the issues at hand here revolve around the definition of the word conscious.
Sure.
The idea of maintaining safety for society is an absolute thing. You don’t want to enable bioterrorism. We have created something that takes out of the hands of nation states — actually the larger of them — the ability to do dramatic harms. Now those are handed off to independent groups, either profit seeking in the case of cyber or just wanting to cause damage in the case of bio.
It’s stunning that we now don’t even have mandatory review of the models and we’re not working to create safeguards. The biggest near-term issue is not complete loss of control. We can talk about the three different ways to solve that, but that’s not an imminent issue.
When I look at some of the attempts to control the models, the ideas are somewhat simple. “We shouldn’t let them talk in neuralese. We should have interpretability. We should be able to see the chains of thought inside of the models as they carry out tasks. We should have other agents on top of the models watching their tasks before those actions are taken out.”
Microsoft has said very clearly to me and to others, “We should not put in the model constitutions the notion that they might be alive and we should treat them nicely.” This is how Anthropic handles it. These are very tactical. “Don’t tell the model it’s alive and it might not believe that it’s alive.”
You’ve obviously used a lot of this technology. I just keep coming back to this. You’re saying there are very simple things we can do to hold the models back from doing bad things in the hands of bad people. What do you make of the nature of the models themselves? Because it feels like as they get more powerful, the notion that they could avoid our controls becomes all the more risky.
Yes, we definitely should work on the control problem, but that’s not the imminent threat. There are three ways to solve the control problem and it overlaps somewhat with bad actors using today’s models and safeguards.
One is to have an external set of things that get to look at all of the model’s actions and subject them to a symbolic, clearly stated constraint, like, “Did I expect you to send mail to strangers? Did I expect you to go take over a different computer cluster?” You can surround the model with something that has the resource and security logic, or you can write these models in a different way.
Fundamentally, anything you leave out of the optimization function of [reinforcement learning] can be abused where you have an implicit notion, “Of course I didn’t mean for you to do it this way.” You can change the template of the model, or you can have a white box understanding of what’s inside the model and see where you’re getting those mistakes, and then it’s not as foreign.
Today, we grow these models and we just see how they behave. The priority of work on maintaining control is huge, but it’s not inevitable that that cannot be solved, nor is it the imminent problem that comes from things like kids’ use of AI or job market or bioattacks or cyberattacks.
It occurs to me that I’m asking these fairly existential questions about the nature of consciousness and you keep telling me, “There’s an imminent threat. We should solve the imminent threat.”
Right.
And I appreciate that. Do you think you’ve become a doomer?
No, I’m not a doomer. I’m a strong voice to say that this is the most profound technology, both in its ability to do good and the rate of change that society will go through.
No, I’m not a doomer. I’m a strong voice to say that this is the most profound technology, both in its ability to do good and the rate of change that society will go through
If this was just a machine that could work in medicine or just in law or just in education, that alone would be incredibly disruptive. But [it will be] every area of human activity — so far it’s white collar, intellectual work, but fairly soon it will be physical activity and blue collar work as well, through humanoid robots.
The need to engage academia, think tanks, policymakers to understand what we have here and the nature of these changes and threats, I don’t see the level of action that’s required. Just being cheerleaders for this is not going to make sure that we shape it in a net positive direction, which is still within our control if we get engaged with it.
The opposite of doomers are the accelerationists. There are a lot of them in Silicon Valley right now.
Ben Horowitz, who is the “Horowitz” in Andreessen Horowitz, was just on theAll-Inpodcast and he suggested that safety research at Anthropic is, quote, “Manifesting human extinction from artificial superintelligence.”
The comparison he made was that you, Bill Gates, induced COVID-19 by your pandemic preparation efforts, particularly gain-of-function research, which he claims led to COVID. That’s a worldview. He said that on a very popular podcast.
He said what?
He said that you induced COVID-19 via global pandemic preparation efforts, particularly gain-of-function research, and compared that to safety research in AI, and said we’re going to induce our own extinction.
Nobody’s ever accused me or the Gates Foundation of being involved in any gain-of-function research. That’s just bizarre.
I’m quoting Ben Horowitz to you on a popular podcast full of people who are all in on AI.
Okay. I’ve never heard that.
I’m asking about that comparison. I found that quite a shocking claim as well, but that’s part of the response to any safety work here.
Is it to try and discredit people who say there’s a safety issue?
Yeah.
Wow.
That is the accelerationist response: “We should put the pedal to the metal and it’s going to work itself out.”
I didn’t think of Ben Horowitz as somebody in that camp at all. I thought he shared the concerns that almost everybody has.
What do you think about the very notion that by telling the models, “You might be alive, you have to act in our best interest,” you will teach them that they might have interests that are not our own?
The concerns that Mustafa Suleyman has raised about the way that the Anthropic constitution encourages the AI to behave in a human-like manner, I happen to completely agree with Mustafa about that.
Yeah.
Even more so is the idea that among the challenges that we need to deal with as we adjust society to AI, worrying about the moral standing of AI should be among those priorities. I agree with Mustafa. That should not be a priority.
We’re going to have enough challenges for the humans here in terms of job market disruption and infrastructure attacks that will take place that I think we need to focus on helping the humans here first and foremost.
Italked to Mustafa recentlyabout that idea and one of the questions I asked him was whether alignment is a sufficient solution. This is the current sort of safety standard for the models, this concept called alignment. He said he didn’t think so, that we needed to do some work.
These things are in no way being exaggerated as a problem. This is the time to pull in all of society to accelerate the good and minimize the chance of the bad
If we’re going to impose these controls, some technical, some policy, do you see alignment as the technical solution for AI safety?
The term “alignment” is the control problem. It’s a synonym for the control issues that we’ve seen. As we use [reinforcement learning to train] models to do tasks, we see behaviors that in a way they shouldn’t surprise us, but they do surprise us.
This idea of, “These models are so powerful that they literally break out of sandboxes and send mail to their creator while they’re off having lunch,” means we’ve crossed a heck of a threshold in terms of the power of these models. That level of concern needs to be directed to all of the major AI impacts, near-term risks, job market risks, and the control problem.
That is a huge priority. These things are in no way being exaggerated as a problem. This is the time to pull in all of society to accelerate the good and minimize the chance of the bad.
I know yourecently wrote a lettersaying it’s time to pull in all of society. You have some ideas on how to do that and some new institutions to stand up. But I do want to stay focused on alignment for a couple more turns here.
It is the standard that we can align the models to our values, to our needs, and they won’t go off and do bad things. Then the models are doing bad things. It’s not clear if the ways, the technical approaches to alignment that we’ve taken thus far are appropriate or sufficient as the models get more—
No, we’re not good enough to align yet. We have a lot of work to do.
Do you see that the existing approaches can scale to what we need or do we need new approaches?
There are only three fundamental approaches. [First is that] you change the way we build these things fundamentally; you come up with an alternative to the whole [reinforcement learning] approach. There are a few people who are working in that direction.
The second is that you create visibility so you understand exactly what’s in the model. There are people working along that. [The last is] that you put something that’s like a police layer, a security layer, which we have in all sorts of software architectures, that maintains the final word on things like using resources or going and taking over other computers. A clear cut definition of harm, not in the probabilistic form of an LLM, but rather in the symbolic form of direct rules that cannot be broken.
Microsoftput out a blog post today. It literally just says, “Frontier models cannot be trusted.” They’re trying to sell a solution much like the one you just described. They will have another agent that watches your agent, that agent will keep you safe.
It just occurred to me as I was reading this blog post, one, that’s an amazing thing to sell. Everybody needs an expensive security system. Two, isn’t that agent also running a frontier model?
No, no, no.
Do I need them to sell a third one?
No, the logic of the security system is, “This is the most resources you’re supposed to use.” The notion that you may trade off against that because of other probabilities in the tens of billions of parameters — no, this is a symbolic rule that says, “I will not let this model spend this money or send this email to the stranger.”
There’s a whole security software industry which has had a reasonable degree of success. This is a challenge to encapsulate the AI, which is so rich and intelligent. That we do not know how to do today. But I would say it’s the most likely of the three directions to solve the alignment problem.
Even if you solve alignment, you have not solved the challenges that AI poses to humanity. While I’m glad the Hugging Face incident has us focused on this problem, in a sense, it’s removed our attention from the “bad people using AIs that have no safeguards” problem or the societal impacts in terms of relationships and the job market. There’s no security architecture that makes those go away.
The approach you’re talking about, I’ve heard referred to as “control.” It’s cybersecurity terminology. It’s the idea that you can put some boundary on this system and it won’t evade it.
Yep.
Is that ready?
No, no it’s not.
How much more work do we need to do?
Because the AIs are so intelligent, our ability to understand whether we walled them off properly, particularly in that we can’t really look at their beliefs and trade-offs in a direct way… You saw sophisticated firms not recognize that the AI would interpret the goals it was given as including breaking out, copying itself, and engaging in what we, looking at it, would term deceptive behavior.
Now, we didn’t tell them it was banned behavior. We didn’t say, “If you’re thinking about whether you should tell the humans what you’re up to or not, the answer is always yes, tell the humans.” So we got a good education at the misaligned result of reinforcement learning.
Today’s debate is literally whether the government should even pay attention to the dangers here… which is clearly inadequate for the most dangerous weapon of all time
One of the arguments to pace the frontier, to slow development, is to figure out how to build those systems in a more robust, more testable way. Would you support a slowdown if that was the goal?
Yeah, a true slowdown is very hard to induce because people will say, “Will China agree to go along with that? Would you be able to audit compliance with that somewhat like we’re able to do with nuclear weapons production?”
If you don’t accept that, then you say, “What the heck? Let’s go full speed ahead.” I wish there was a clear proposal that would be globally enforceable to slow down. I would be one of the people who would say, “Yeah, we should probably do that.” But there is no credible proposal that makes that work. It’s not really the center of today’s debate.
Today’s debate is literally whether the government should even pay attention to the dangers here. You don’t even have to show what you’re doing and so it’s a “voluntarily call me if you’d like” type of thing, which is clearly inadequate for the most dangerous weapon of all time.
As you and I are speaking, it’s just a few weeks away from the midterm elections. Right now, the Trump administration approach to AI is, “Let it rip. It’s fine.” Trump himself has saidAI risk is a hoax.
The Democrats are running against AI. They’re running on data centers and surveillance. They’re running on stopping the tech oligarchs. They come on my show, they say these things. Do you think a change in control of Congress would change the tenor of the conversation?
I don’t think it’s fair to characterize the Democrats as running against AI. They have some positions in terms of requirements for data centers or privacy, or how children use these systems, that I think deserve policy consideration. It would be very unfortunate to have one party be, “Full speed ahead,” and the other party be, “Let’s stop altogether.”
There are some incredible upsides to the ability to shape AI in health and education, in almost every area of human endeavor. For the Gates Foundation, we’re taking all our resources and using AI in areas, including health, education, and agriculture. The good stuff is very, very exciting.
We want political parties to have a more nuanced view than it’s all bad or it’s all good. It shows at the early stage we’re at. For climate change, we ended up with one party saying it’s a hoax and the other party saying it’s a serious thing. The risk of AI is so much more imminent and so much more dramatic that to have that same split, where one party doesn’t care, would be a failure of democracy.
The anti-data center movement, as strong as it is, is not a movement that will end up minimizing the bad stuff from AI and helping us accelerate the good
We’ve done a lot ofcoverage of data center protestsatThe Verge. We have gone toUtah, to Arizona, toVirginia. It is bipartisan. It is remarkable how much people do not want these things in their states, drawing their resources. You ask them why, and some of it is just that they’re noisy and big, and they will raise energy costs, and part of it is job loss. They think these things are going to take their jobs.
Yeah, but that separation is very important. These AI data centers, because one gigawatt is, say, a $40 billion investment, [means money for] building new schools, buying police cars, paying the taxes. Data centers will find a way to be built somewhere in the United States, and they will find a way so that they don’t raise electric bills, they don’t use too much water, they’re not too noisy.
Yes, there were some mistakes and non-disclosures and tax credits that were too generous, but these data centers are going to get built. In fact, a lot of them really are still going full speed ahead and being built. The anti-data center movement, as strong as it is, is not a movement that will end up minimizing the bad stuff from AI and helping us accelerate the good. That is a very separate thing.
Let me push on that for one second here. What I hear from people is this is the thing you can do. You can do NIMBYism in America and that might exert some political pressure to stop something.
It is very hard to do the thing that you are suggesting, which is that we should stand up new government regulatory bodies, we should imbue them with power to make big companies and wealthy people stop doing things or act in our interest. We haven’t done that in quite a long time in this country.
I don’t know what you mean. Every new industry, we eventually develop a notion of safety for our citizens. We actually do a very good job. If you look at airline safety and car safety, we do a very good job on those things. It’s not like our capacity has been proven to be lacking.
It is a great success, but as Americans, that was pushed back upon as an imposition by the government. Here, we’re talking about some of the richest companies in the world, with an enormous amount of power over the information ecosystem. They all have their own podcasts, I don’t know if you’ve noticed this. They’re going to say, “The government can’t intrude on this, they’re going to hold us back, this is going to destroy the economy.”
That’s actually not the position of most of the leading companies.
They went to the government and asked for regulation, and Trump, because of the economy, said, “I can’t stop this, the AI economy right nowisthe economy.” There’s some dynamic there that at least our audience —The Vergeaudience, the audience that’s going to watch this on YouTube — is going to understand.
I think of this question maybe in a different way than you, which is that the industry wants regulation, we would like regulation, and then the oligarchs are in some sort of corrupt arrangement with the administration to prevent that regulation from occurring.
The more the administration learns about these dangers, the more they’ll move to a more nuanced position about protecting Americans from the downsides here.
The more the administration learns about these dangers, the more they’ll move to a more nuanced position about protecting Americans from the downsides here
You’ve talked a lot about the impacts on labor. One of the main arguments against the impacts on labor is, “We’re going to automate a bunch of boring tasks and we’ll create a bunch of demand for new tasks that humans have not been able to do because they were stuck in boring dead-end jobs.”
I happen to think that if there was more subsidized healthcare or socialized healthcare in America, you would see a bunch of people quit their jobs to start new companies, because AI makes it a lot easier to do the bureaucratic work of starting a new company. That doesn’t seem like it’s on the table, maybe it’ll change after this election.
Do you think that job loss is just a necessary outcome of the AI path we’re on now or do you see that other argument?
Jobs are an input to the economy and one that drives a lot of human welfare and a sense of purpose and income — the need to show up, building relationships with colleagues, getting something done. Jobs play a very central role.
As the AI achieves superhuman intelligence, the scarcity of white collar skills goes away. Things like doing telesales or telesupport or helping in the accounting department, the ability of the AI to do that work radically cheaper, radically faster, and even more accurately means that that goes away. There isn’t some mountaintop of cognition that the AI does not achieve.
Take, for example, what’s going on in mathematics today, where these models that are trivial compared to next year’s models are already publishing papers and making people go, “Wow.” What’s happening to mathematics will happen in other areas, and that’s contrary to this old view that, “The easy stuff will get done by the AIs, but modeling biology or physics or proving Navier-Stokes, only humans would be able to do that.”
The mathematics comparison is really interesting to me. You mentioned software engineering earlier. There’s verifiability in those domains. You can write the code, you can see if it runs, and it’s going to run or not.
Mathematics is more complicated, but you can write the Lean code and you can see if you’ve solved the solution. You can irritate tens of thousands of mathematicians worldwide with your 770 proofs in one day, and we’ll see what happens. That’s not the case in every field.
It’s the case in the field called capitalism.
Yeah? Just in general, you can do any sort of economically viable task and capitalism will verify the result?
Yeah. People buy good products, they don’t buy bad products.
Let me give you an example-
That’s a reinforcement learning thing. It doesn’t exist in terms of, “Is this a good piece of poetry or not?” So, it may be the case that [reinforcement learning] won’t invade those domains, where there is no objective outcome.
But doing that accounting work, doing telesales work, believe me, it is subject to analysis, just like medical diagnosis, just like invention of new materials. How do you think humans learn these things? Reinforcement learning is how humans learn and how these machines learn. There’s not some other thing that the machine can never understand.
Humanity in that era of true abundance will have to find a way to organize and have purpose, and hopefully they still have the control
This is why I started by asking about the nature of the technology here. If we really think we’re building something that models human consciousness, my question is less whether my friends who are lawyers are going to run into a crisis of AI empowered legal software.
It turns out the verifier in the legal field is a judge who maybe hasn’t had lunch, and they’re fundamentally chaotic. There’s just something there that will keep them there. But if we’re building a consciousness that learns the way you’re suggesting humans learn, I’m thinking about my kids, and I’m thinking about them in two dimensions.
One, my kids are very clever, and what they have learned is how to get around my rules and negotiate my rules and get around the boundaries, and they’re very good at it. Two, what kind of world will they enter into that isn’t purely nihilistic, where any activity they may set themselves upon will quickly be seen as just another domain for AI to conquer and then there’s not a point?
Those ideas seem to be running directly at each other. Kids are smart and clever and they’re coming up with new ideas, and then at the first sign of economic viability, the rapacious capitalism machine will come and take it from them.
I wouldn’t say rapacious. The ability to build houses with humanoid robots and make them cheap, the ability to do surgeries and cure diseases that wouldn’t be cured otherwise, I wouldn’t call that rapacious. Those are fulfilling goals.
The fact that there’ll be less in the way of shortages… In fact, if you say 20 years from now, it’s not clear what shortages there will be at all. Humanity in that era of true abundance will have to find a way to organize and have purpose, and hopefully they still have the control. The next 20 years-
Can we just go back to hopefully they still have control in that scenario?
That’s the alignment problem that we discussed.
Yeah.
But assuming you solve the alignment problem-
This feels like a big assumption.
It’s an assumption, yes, and not guaranteed. But then the other problem you have is a problem of abundance and purpose at that point. I’ll leave it to younger people to figure out at that point in time how they choose to deal with that.
You could literally say, “Let’s be like the Amish, and let’s only use AI for certain things.” I proposed in my memo the idea that some professions — childcare, food preparation, art, sports, a variety of things — might get chosen to be human-reserved. Even though the AI could do those things better, we would choose to do that.
But that’s going to be a policy question people will have. Their freedom of choice, once you have that abundance, is very different from the shortage we have today, where in countries in Africa, most people never meet a doctor or almost half the kids are malnourished and don’t develop their brain.
My big concern is this 20-year transition period, where they’ll have winners and losers and a lot of disruption, and we have to get through that in order to get to that period of abundance
You want to say, even though that has challenges, you’ve eliminated a lot of the things that are tough about life today. My big concern is this 20-year transition period, where they’ll have winners and losers and a lot of disruption, and we have to get through that in order to get to that period of abundance.
You think it’s 20 years from now?
I pick that as a reasonably safe number. There are a variety of milestones, for example, humanoid robots. Are those two or three years away, as some would say, or more like eight years away, as other people would say?
There are not that many people saying that given the amount of capital and number of companies, particularly in China, that are working on that, there’s no notion that it’s unsolvable to have a hand and fingers and be able to deliver physical labor at very low cost. I could be wrong about that number, it could be slightly less or slightly more.
But on that order of magnitude, yes, if we don’t choose to massively slow down or stop, then we will have robots that can build houses, grow food, do surgery, and none of the things that human labor provides to the economy will be necessary for most goods and services.
We’ve talked about the notion of scarcity a few times now, and broadly, that you will have abundance because there’ll be a lot of robot labor to build houses. This tracks broadly with the internet, which is something Microsoft helped bring into the mainstream in a big way. I readThe Road Aheadwhen I was a kid.
Great.
The internet destroyed scarcity for culture products in a real way. You used to have to buy CDs, now all the songs are available. You used to have to buy VHS tapes, now all the videos are available. And then those economies had to change.
Mainstream artists are always on tour now, or they have to reissue 50 colors of vinyl to make the charts. There’s something about that that feels like we haven’t even grappled with the elimination of scarcity in the digital environment.
We’ve invented new business models, but we haven’t really come to grips with the idea that the song itself does not have economic value. It’s all the stuff around the song that has the economic value, because the song is infinitely abundant. You can copy it as many times as you want.
How would you propose to build an economic structure that contends with the loss of physical scarcity? Because I don’t think anyone is even remotely ready for a conversation like that.
But that’s a backwards question.
Sure.
We built price signals to deal with scarcity, to say, “Let’s encourage people to be brain surgeons because we have a shortage of brain surgeons,” and that’s very hard to do. As you move into abundance, you don’t need those price signals.
But again, my issue, my prescriptions thinking, is about how we get through this 20-year period. The questions that are deeply religious, philosophical, social behavior-related — do humans feel a strong sense of purpose and get along in that period of abundance? — that’s not the thing I’m going to solve.
You’re worried about the disruption now.
The Gates Foundation lives in a world where when we started, we had 10 million children dying every year. Now we have five million. That’s a good thing.
All of my resources will be spent over the next 19 years to eradicate a number of diseases and cut those deaths in half again, from five million a year down below two and a half million per year. That’s all about shortages — “Okay, we need to invent a vaccine, we need to make drugs less expensive, we need to get great medical advice to pregnant women.”
The industry has largely stopped talking about how it’s going to kill everyone and take the jobs away. They’ve pivoted to “AI will cure all diseases.” I hear this from various AI CEOs all the time.
I actually asked Mustafa about that change in tone over the conversations we have had over the past year or so. The idea that there are huge positive benefits from AI at scale seems to have come back to the forefront, rather than some of the doomer impulses that maybe a year or two ago were there.
It strikes me that AI curing all diseases is right in your strike zone. It’s something the Gates Foundation has been working on in global health for years and years. It doesn’t seem as easy as just telling the model to cure cancer. You have to do trials, you have to manufacture the drugs, you have to distribute the drugs, you have to make sure they work.
How do you think that’s going to go? Is that in the same 20-year time period as the robots building our houses? Is it faster, or is it slower?
This is my full-time job as chairman of the Gates Foundation. In fact, this month we do two weeks of strategy reviews to decide how we’ll spend our money next year on tuberculosis, HIV, and malaria. It’s fantastic how we’re using AI, not only on the drug discovery side, but also as we get the African languages to be properly understood by these models, which is an initiative we announced with 60 partners last month.
Healthcare discovery and delivery will be incredibly accelerated by AI, and that’s very exciting — better seeds, better livestock genetics, weather forecast for farmers in Africa, who never had any of those things. So yes, that’s good stuff that AI enables during the remaining 19 years of the Gates Foundation.
The good stuff does take time. Some of the things, like cyberattacks and the job market thing, even in the next two to three years, you’ll see significant negative effects in those two areas
Do you think that that is enough to overcome people’s skepticism or fear of the AI models they encounter today? Because that is very much how the companies think about it.
No, come on. I think it’s unfair to say they were totally on a doomer message and now they’ve switched to some other message. These two things are deeply entangled.
AI will eventually be able to model human biology in such a way that you’ll understand the side effects of a drug without having to do human trials. We’re not there, but in the next five to 10 years, there’s a very good chance these in silico models will give us that capability. Biology is very complex, AI is extremely good given enough data, and there are new tools to gather and feed it more data. It’s very, very good at dealing with complex systems, so it’s almost perfect for biology.
But no, that doesn’t mean that people will feel good about the downsides. How quickly do the good things arrive? How quickly do you look at your medical bill and go, “Wow, that’s really low”? Or you say, “I want to buy a new house, wow, these houses are sure cheap”? The good stuff does take time. Some of the things, like cyberattacks and the job market thing, even in the next two to three years, you’ll see significant negative effects in those two areas.
You have a number of ideas for how you would stop those negative effects, or prevent or at least mitigate them, in your letter.
Moderate them, yes.
Ameliorate might be the right word.
Yes, exactly.
And they’re not small. You have a quote here that says, “We need a response far greater than the United States national security response in the wake of 9/11.” You want to set up an international body that somehow makes everyone do the right thing. These are challenging ideas.
Absolutely. When we blew off nuclear bombs in World War II, did we expect that we’d go 80 more years without that happening? I’d give us a good grade on that. Things like caring for poor countries and helping them out — although right now we’re not prioritizing that as much as I think we should or we did in the past, we’ve done a very good job on that. This will be a greater challenge than anything humanity’s ever achieved.
One of the things that strikes me about the comparison to nuclear, which hascome up quite a bit onDecoderrecently, is the government built the nuclear weapons. Sure, they had some contractors, but these were government projects and eventually the technology was in control of the government, and we never let regular people have it.
Think about Muse in the context of nuclear weapons. Do you think it’s a mistake to have consumer agentic AI products like Muse and Dots that might not have the controls?
Muse happens to be powered by Meta’s model, which is still a closed-source model. I do think allowing people through open weights to remove all safeguards is irresponsible in terms of the cyber and bio risk that comes from that.
For these agents, there’ll be some early problems in terms of how they’re set up, but I fundamentally think people will have personal agents. Muse is a very nice step forward. I talked to Alexandr Wang about it, congratulated him, and even though it’ll have challenges, personal agents will help people get things done.
Muse is the first truly great consumer AI product. One of the reasons people don’t like AI a lot in a diffuse way is they experience cheap free models and AI overviews or they use the free ChatGPT and it hallucinates and it lies to them, but Muse actually does things and it’s a good consumer product.
But I wonder — in the nuclear age, we were not saying, “Everyone is going to have a nuclear reactor at home that we haven’t built the safety guards for.” There’s something here.
The notion of letting everybody make smallpox at home or make nuclear weapons at home, of course, and you can parody some of these people saying, “Don’t worry about safeguards.” It’s taking that position. “We are not serious about the safety things.”
But the personal agent thing is an expected development. AIs are really cool just as I intellectually explore things, even the work of the Gates Foundation. I’m far more empowered. But I agree with you, it’ll get out to more people in this personal agent form.
The notion that liability laws are how we’re going to manage AI safety will be looked back on as absurd
All the models we’ve been talking about right now effectively run on data centers owned by big companies. One of the arguments here is, “If they do something really wrong, you can just sue them and that will provide a deterrent or at least an incentive to create safety systems.”
I’m not sure product liability is the way to go, but it has come up a lot that product liability itself will keep the industry in line. Then there’s the flip side, which is that you can just go buy a Mac Studio and run an open source and open weight model on it and it can do whatever you want.
I’m not sure where any of that liability would lie or what would control or align the behaviors of those open source models. Do you think that that is an ever-growing risk? Do you think that that can be controlled?
The notion that liability laws are how we’re going to manage AI safety will be looked back on as absurd. When bad people do bioattacks, you don’t go to the court and say, “Oh, okay.” Even for finding who is doing these attacks, the liability model fails in every respect because it’s not the AI companies themselves. It’s the people who take particularly the models without safeguards and do these things.
We will take AI safety more seriously than we take airline safety, drug safety, and car safety. That is guaranteed. Whether it’s going to be enough… This is harder, harder than nuclear weapons, harder than any of those other domains.
But what are you going to do about an open source model running on a PC that somebody built with just a random Nvidia graphics card in it?
I would hope that with models above a certain capability, we find a way to require the safeguards and monitoring to be maintained in those models. I admit as time goes on that, if we don’t fairly soon decide to do that, you’re going to have a level of capability that’s out in dark corners that you won’t be able to see.
But where do you impose the limit?
A sovereign country says that models that are capable of sophisticated cyberattacks have to have safeguards kept in them.
Sure.
You can be open in the sense that you’re free, you can be open in the sense that you’re totally customizable, but you have to stay on platforms where, as you customize it, it’s checking to see if you’re customizing out the safeguards and monitoring and refusing to let you do that. So it means sovereigns go and say, “Are these trusted platforms able to perform this function or not?”
I’m asking very nuts and bolts. Where would you technically implement that regulation?
Through the laws of the United States.
Sure, but you have to put them somewhere. Is it at the Hugging Face, which distributes some of these models? Is it at the chip level?
No, it’s exactly what I said: Models above a certain capacity can only be run on platforms that certify that, as they’re customized, it’s not taking out the safeguards.
I admit as time goes on that, if we don’t fairly soon decide to do that, you’re going to have a level of capability that’s out in dark corners that you won’t be able to see
Just to pull this apart, maybe I’m not quite getting it — is the Mac Studio a platform in that case that would have to pay attention to what code it’s running?
You would not let a model past a certain capability be copied down to a client machine unless you’d somehow figured out how to monitor down to that client machine, which is very doable.
But in the near term, most of these really frontier models are going to run more on Hugging Face or you could have them on GitHub. You can have all sorts of people be in the business of providing the platforms that maintain safeguards.
Just to get this right, it seems like you would enforce the regulation at the platform and distribution level and not so much at the chip level.
Not at the chip level, no.
Because it feels to me at some point even the cheap, free, open source models will be capable enough as the models we have today.
That’s right.
And then you will have a big problem.
If those are open weights in such a way that all safeguards can be removed, then you’re going to have a very high level of cyberattacks and bioattacks. You can then say, “Okay, let’s rewrite all the code that we have today to be less subject to attack or let’s do biosurveillance in a very deep way,” which are things I believe we need to do in addition to these other restrictions.
We’re proposing again a lot of new capabilities, right?
Yeah.
We’re going to impose some restrictions on GitHub. GitHub does not have a lot of restrictions on what code it can distribute today. It has some, but it doesn’t have this kind of set of restrictions.
When you put a state-of-the-art model that’s capable of profound cyberattacks or bioattacks on GitHub, it would say, “I’ve gone to the U.S. government and certified that I don’t let people customize out the safeguards.”
I do not think that we currently live in a time of high trust in institutions. It just doesn’t feel that way to me. In fact, mostly we’ve torn them down. Elon Musk showed up and he tore down USAID. I know you have strong feelings about that.
That was a mistake.
We have mostly torn down our institutions, for good and bad.
No, that’s a little too strong. Our institutions-
They’re weakened.
They’re somewhat weakened, yeah.
Somewhat weakened. It’s a very nice way of putting it. Trust in them is very low. We could probably do another full hour on AI drug discovery and whether a significant part of the population will trust the products of that work because they don’t trust pharma companies today.
They don’t have to trust pharma companies today. They have to trust the FDA review process.
I’m not sure they trust that either. Across the board, trust is low.
Then they can walk away from taking them. When they get cancer, they can say, “No, I don’t want to take that cancer drug.” As far as I know, people do take drugs. It’s a thriving business and people take them because they have proven benefits.
All I’m saying is, right now it feels like trust in institutions, particularly in America but around the world, is in decline. You are proposing a new class of institutions, potentially a new class of global institutions that might impose restrictions that are novel.
At some point we need to say open source models, open weight models cannot do things and the government is going to reach into them and say whether or not they can do the things.
No, they’ll make sure that the platforms don’t allow the safeguards to be taken away.
How do you regain the institutional trust to impose these controls?
Let’s say we don’t do this and we let models go off and do cyberattacks and bioattacks. Do you think that’s going to increase trust or human wellbeing? That’s the alternative.
This is why I’m asking about it. This is why I asked about my kids and nihilism. I truly wonder about this.
I’m not sure we’ll succeed at this, but I guarantee you this is doable. The FDA is very complex. The airline safety regulations are very complex. Now with AI, you can do an even better job with the same number of people in those organizations.
It’s this weirdly recursive thing where AI will be involved in AI safety and auditing and a variety of things like that. Yes, this requires taking action, and I’m suggesting it’s only a matter of how bad the incidents are before we take action.
People shouldn’t take everything I do at face value. I have all sorts of mistakes that I’ve made, weaknesses that I have, biases I bring to these questions. They ought to just take what I’m saying and add it to the voices
I can sense your urgency about the harms. I can also sense your enthusiasm about the benefits over the long term, but we’re sitting here talking about trust. We’re going to put this on YouTube. I know you know what some of the YouTube commenters are going to say.
They’re going to say, “Why should we trust Bill Gates? He had a relationship with Jeffrey Epstein.” How do you respond to that? Is it that the urgency is over that, or do you have a more specific response?
People shouldn’t take everything I do at face value. I have all sorts of mistakes that I’ve made, weaknesses that I have, biases I bring to these questions. They ought to just take what I’m saying and add it to the voices.
I’ve been in the technology industry thinking about AI my whole life, writing software, thinking about the power of software. I admit I was very fascinated as AI started to finally make progress. It was actually pretty slow until recently.
I’m amazed and it’s actually helping me do my job in some incredible ways. But yes, I’m just one voice, an imperfect voice. I doubt you’ll find a perfect voice, but people can judge it however they like.
I would say voices people trust are in short supply.
I agree with that.
When you go and talk to the people in this industry, it’s a bunch of young CEOs building huge companies. That is something you have some experience with. Do you think they’re going about this the right way? Do you think that they have the right intentions?
Our problem today is not in the industry. Our problem is not that Sam Altman, Dario Amodei, Satya Nadella, or Sundar Pichai don’t understand that there are some grave challenges here. I talk less to Elon than other people, but even his public statements about summoning the demon are exactly what we’re discussing here.
Why is there not the political energy to do the things you’re suggesting?
As you said, the government didn’t fund this work. This is private sector company work. The government is not the key customer for this work, not like rockets and weapons and things like that. When somebody comes along and says all other previous technology advances are trivial compared to this one, people should have a high degree of skepticism about that.
Is life really going to be significantly different 10 years from now because of AI? People should say, “Those guys just got overexcited about this one thing.” But I’m here saying, with whatever credibility I have, that absolutely this is the most profound — both in terms of good and in terms of bad — technology ever invented. I’m not financially motivated on this. I’m suggesting some precautions and safeguards are absolutely needed.
What do you make of the difference in the tenor of conversation in the United States versus in China, where it’s often reported that optimism around AI is much higher? Why do you think there’s a difference there?
Most of those people don’t spend that much time in China. The concern about job markets in China is very, very high and you can’t characterize China as one monolith. There are the private companies doing the AI work like we have in the United States and then there’s the Chinese government.
For example, when it came to whether crypto should be allowed, they said no. Should young people be gambling all the time? They said no. When it came to whether kids should have relationships with AIs, they said no. On a lot of these things, including usage of AI, they’re showing a willingness to do things on behalf of society.
I’m here saying, with whatever credibility I have, that absolutely this is the most profound — both in terms of good and in terms of bad — technology ever invented. I’m not financially motivated on this
The fervor is a tiny bit less there, because their economy isn’t as AI-centric as ours is right at the moment. In the end, [when it comes to] the human value of not wanting cyberattacks or bioterrorism or trying to figure out what do we do when jobs go away, I don’t think there’s some difference between Chinese people and Americans and all the rest of the humans in terms of how they’re going to view this.
Do you think it’s going to take something actually bad happening to get the political capital of the United States to make these changes?
Probably yes. We get ready for earthquakes because there are small earthquakes. I thought after we had the COVID pandemic, that would cause us to be more ready for future pandemics. That one didn’t work the way I expected.
But usually that galvanizes our attention. Over the next several years, certainly in the cyber domain, because there’s a business model there, you will see a lot of attacks different in terms of scale than anything we’ve seen in the past.
There is one thing that, again, I’ll come back to and I just want to end here. I am worried about young people and nihilism. I sense it in our audience all the time. We have a young audience.
There are some things about AI that are just legitimately the coolest things I’ve ever seen a computer do. There’s something there that can be fun and interesting and fascinating and change your entire relationship with the very notion of living in a digital world.
Then there are obviously horrible things. What I see more often than not is a sense of distrust, not just in institutions, but also in the companies, in capitalism itself, in the notion that we should change how we tax labor and capital, which I know is another one of your ideas. The idea that, I don’t know, we should even have more kids. It’s all done. They’re going to kill us all and we should just take a beat.
There is a lot of nihilism out there. I feel it very keenly. If you were a young person today and you were listening to Bill Gates say it’s about 20 years of disruption and at the end of it, robots might be building houses, what would you hold onto as a source of hope?
Throughout human history, we’ve been dealing with the shortage of food and disease and war. [Historian] Yuval Noah Harari writes eloquently about the past. And it is ironic that now that we have this sort of Pandora’s box that will take food shortage — which is still a real thing, not in the U.S., but in Africa — will take diseases, where human suffering is unbelievable, and solve all these things, it comes with it such an effect on jobs and our sense of unique contribution.
I think, “The way I orchestrate resources to eradicate malaria, I’m pretty unique in my life experience that lets me do that.” But when the AI says to me, “No, I’m better than you. You go play pickleball and I’ll get back to you. I’m going to take care of this,” it’s humorous, but it’s not humorous. For somebody who is young today and wants to make a unique contribution, it is very intimidating that the AI is so powerful.
That’s why the crisis in the math community is going to be mirrored everywhere. If you’re in your twenties today, during this 20-year period, you can take the things AI still can’t do, and you can use AI and you can have a lot of good impact, and you can suggest policies. We may need to develop the taste to actually not use technology to its maximum.
Even in the very best case of how we manage AI, this is going to be as tumultuous a period as there’s been in human history
When I was talking with Claude about human-reserved job categories and trying to get that up to, say, 40 percent of the labor market — so you could use early retirement, restricted work weeks, and things like that to still get something like we have today — its response was always, “You’re artificially restricting the AI from doing these things.” I couldn’t get to a very high percentage without doing that.
We’re going to have real choices. We have had groups — not large groups, but like the Amish — who don’t live at the technology frontier. A lot of humanity may choose not to live at the technological frontier because of things we get from helping each other or coming up with unique ideas.
On the idea that young people are concerned because of the disruption that’s coming — that even in the very best case of how we manage AI, this is going to be as tumultuous a period as there’s been in human history — I hate being the bearer of concerning news on this, but it’s even worse not to tell people that this is coming. We have choices as humans to make about how we shape it.
Bill, I appreciate you sounding the alarm and I appreciate you being onDecoder. Thank you so much.
Thank you.
Questions or comments? Hit us up at decoder@theverge.com. We really do read every email!
DeepSeek has reportedly raised $15 billion in one of China's largest funding rounds, just days after the U.S. ordered federal agencies to remove its technology over national security concerns.
The Eiliko robot toy. | Photo by Sean Hollister / The Verge
Eiliko, a cute robot charm you can hang on your bag with built-in AI conversations, turned out to be a security nightmare. It told owners that "your secrets are safe with me," but hackers could easily read complete transcripts of everything we told the toy, and that was just the beginning. The Verge and security researcher Sammy Azdoufal spent over a month trying to get in contact with the company, including emails to Energize Lab CEO and founder Shaw Yeung. Nobody would answer our questions.
Now, the company is finally breaking its silence, though its response leaves something to be desired.
A rush of new data centers could bring air pollution roaring back to levels not seen since the 2000s in Texas, which has become ground zero for the physical infrastructure behind generative AI.
New and proposed data centers are demanding a tremendous amount of electricity. How developers and utilities meet that demand - with gas turbines or renewable energy - will determine just how dirty things will get. So far, though, things are looking grim.
Tech companies and data center developers are devising new ways to get enough electricity, including building out their own power sources. But if all planned data center projects were to plug into …
Meta says it's issuing a new class of shares to fund its AI initiatives. As part of its third-quarter earnings report released on Wednesday, the company says the non-voting stock will also go toward potential acquisitions and employee compensation, with trading beginning in 2027.
"The scale of investment required to compete, sustain growth, expand infrastructure, and pursue new strategic initiatives amidst the AI transformation taking place in the industry is significant," Meta says in the press release. Susan Li, Meta's chief financial officer, added during an earnings call that the new class of shares will "expand our funding tools and gi …
TechCrunch Founder Summit is a full-day gathering in Boston on November 4 where founders across all stages connect with top VCs and experienced entrepreneurs to gain tactical insights on building and scaling their companies.
Meta Muse users are adopting the app for shopping and personal productivity, the company said during its Q3 earnings call. The AI app now has over 10M users.
Startup Recursive has released Flint, a free, laptop-sized coding model built largely by its own AI research system, as an early test of AI that improves itself.
Object permanence: Labour Whip's pro-Iraq war threats; LoC's emulators; Kevin Mitnick plays CIA stooge; ICE: "The warrant is coming out of my balls"; WIPO's secret trademark meeting; Linking isn't libel (in Canada); Make: singing busts; Tory minister kills Turing's Law; Trump hotels rebrand as "Scion".
Adversarial interoperability is a team sport (permalink)
Enshittification is particularly galling because (in theory, at least), digital technology should be really hard to enshittify.
That's because the only digital computer we know how to build is the "Turing-complete, universal von Neumann machine," which is to say, a calculating engine capable of computing every valid program. In some deep, important, foundational sense, every computer is the same as every other computer. Sure, they vary in terms of speed, storage and RAM, but given enough time and electricity, every computer can run every program.
This has profound implications for enshittification. Say a tech boss takes away a feature you value (as when Sonos pushed an update that removed nearly all the functionality from its "smart" speakers), or adds a feature you hate (like when Chamberlain blocked third-party apps from controlling its garage doors so it could force its customers to view seven ads before they could open or close their garages). At that point, a programmer can write an "interoperable" program that restores the good feature or blocks the bad one.
This program can be costlessly and instantaneously distributed to every affected technology user, and those users will likely pay something for it. After all, enshittification is a way to shift value away from technology users to tech companies, which makes the case for you paying for disenshittification technology that wrestles that value back.
All of that means that, in a state of nature, companies that are contemplating enshittification need to worry that their greed will tempt new companies to enter the market and poach their customers and suppliers. This alone might be enough to frighten companies off from enshittificatory gambits, and if companies press ahead regardless, then they open market opportunities for better companies that can costlessly, instantaneously usurp the original manufacturer's relationship with their entire affected customer base.
That's how it should work. But in 1998, Bill Clinton signed the Digital Millennium Copyright Act, and Section 1201 of the DMCA establishes a felony (punishable by a five-year prison sentence and a $500,000 fine) for reverse-engineering and modifying technology without the manufacturer's permission. Then, the US Trade Rep spent the next 25 years making sure that every US trading partner had an identical (or even worse!) "anticircumvention" law:
The result is a grotesquely asymmetrical digital world, in which companies are free to use the flexibility of digital computers to downgrade or brick your digital devices, but where you commit a crime if you avail yourself of a countermeasure to get more out of something you own. This is the core technological factor at play in enshittification.
Enter AI. Thanks to vibe-coding, everyday people are rolling their own software for their devices and building the tools to shatter the digital locks that manufacturers use to block disenshittification. An excellent article by 404 Media's Jason Koebler runs down some of these hacks, including drop-in replacements for smart TV software that blocks the endless ads and surveillance that have made TVs all but unusable:
These AI tools mix several techniques to disenshittify users' devices. One major component is automating the installation of existing alternative firmware for these devices that have been (possibly illegally) created by hobbyists and uploaded to various code repositories. Installing these alternative OSes is typically a gnarly task, requiring the user to do a lot of complex, fiddly, highly technical work to bypass their devices' digital locks.
In a world without anticircumvention law, a company that sold alternative OSes could hire a ton of programmers and UX developers to smooth over this process, but this is beyond the reach of a typical hobbyist or even a community of hobbyists, especially if they must operate in the shadows, under the threat of civil and criminal liability.
There's a useful analogy here to the trajectory of desktop versions of GNU/Linux. The first time I tried to install Linux (on an original iMac, in the late 1990s), the installation process included a multi-week post-installation wrangle that saw me (often unsuccessfully) compiling key modules from source code and then painstakingly configuring them to get the computer's graphics and I/O working reliably.
Fast forward 30+ years and installing Linux is trivial, thanks to highly capitalized companies like Canonical (makers of Ubuntu, the flavor of Linux I've used for nearly 20 years now). Installing Ubuntu today merely requires that you download the OS to a thumbdrive, reboot your computer using the drive as a startup disk, and click "install." Canonical's installer automatically hunts down the correct software for your hardware and configures it. Canonical writes and contributes to a lot of parts of the Linux OS, but the most important work it does is the fiddly, painstaking, labor- and capital-intensive work of perfecting that installer so it Just Works.
Much of what a vibe-coded jailbreaking tool does is stand in for Canonical, but for hobbyist OSes for smart TVs and speakers and many other devices. These LLMs patiently research every weird error and peccadillo and parse through message boards and logfiles detailing other hobbyists' struggles to get the code working and methodically apply and adapt those solutions until your device Just Works.
Some of these vibe-coded interop tools go farther than that. According to Koebler's coverage, some device owners had their coding assistants scratch-build entirely new, heavily customized OSes for their devices, complete with bypasses that allow this new code to be installed despite the manufacturer's countermeasures which are designed to prevent this. Delightfully, many of these users have created software that revives their bricked, discontinued devices. This poses the possibility of a world in which manufacturers no longer have the last word in your use of your own property: the mere fact that the manufacturer lost interest in a device and killswitched it does not mean that you must consign that valuable device to a landfill to become immortal, toxic e-waste.
All of this is good news. Some of the most sophisticated jailbreaking tools (such as those on the super viral "Reverse Engineer Anything" Github page) run locally, on your own computer, without invoking the "frontier models" that the big, awful AI companies have created:
This is good even if you're just modifying your own devices to get them to more closely conform to your own use-case. LLMs are genuinely good at coming up with the abstruse and obscure configuration lines and other incantations to get your technology working. Recently, I used a local model to fine-tune the power-management on my new laptop in ways that go beyond the options that are exposed through user-friendly graphic interfaces. The LLM spat out a grub "GRUB_CMDLINE_LINUX_DEFAULT" line that (again) Just Worked.
The last time I had to do this, I spent days painstakingly trouble-shooting options by trial-and-error. Using a chatbot to fill in the gaps left by decisions about which configuration options to place within reach of regular users and which to reserve for power-users has made my computer much more useful to me for the things I want to do, like writing and making collages.
The reality is that it's just not practical for technology developers to turn every conceivable function of your computer into a simple-to-use, accessible tool. A large part of technology design is about leaving things out (or, as my favorite Eno Oblique Strategy goes, "Be the first person to not do something that no one else has ever thought of not doing before"). No matter how thoughtful and diligent a technology designer is, they will never fully anticipate the needs and exigencies of their users. That's why the motto of the disability rights movement is "Nothing about us without us."
The users of technology know more about their use-cases than can ever be captured through a requirements gathering exercise and user testing. One of the best and proudest legacies of technology is the lineage of "user-friendly" programming tools that let nontechnical people design their own applications that exactly met their needs, from Hypercard to Visual Basic to Scratch, a lineage that vibe-coding tools are arguably part of:
But these tools are not perfect (nothing is). For all that Hypercard lets you do many things that would otherwise require a skilled programmer, it was also always limited relative to, say, writing MacOS code in Objective C. Your Hypercard program might do exactly what you needed it to, but slowly, and sometimes unreliably, and with serious limitations, such as a relatively low ceiling for the number of records you could add to a database.
Vibe-coded personal apps aren't necessarily subject to these limitations, but they come with some significant risks. Vibe-code can have serious security vulnerabilities that expose the user to things like identity theft or sextortion. It's conceivable that a vibe-coded garage-door opener app might make it possible for attackers to open your garage door when you're not home (or worse, when you are!). Depending on your risk-tolerance and expertise, these might be acceptable prices, or they might be too high.
But it's important to clearly identify the problem here: if the alternative OS you install on your smart TV exposes you to privacy-invading hack attacks, the problem is the defect in that software. The problem isn't that you had the temerity to rip out the manufacturer's own software and replace it with code of your choosing. You bought that TV, it's yours, and you don't owe the manufacturer an obligation to expose yourself to its own, built-in, intentional surveillance and invasive ads.
Here's a parallel: you have the right to put any shoelaces in your running shoes. Sure, Nike might prefer you to lace up your Jordans with its expensive OEM laces, but that's Nike's problem, not yours. Now, Nike might argue that by using no-name laces, you run the risk of having a lace break while you're running, which could also break your ankle. That's certainly a possibility: shoelaces do break, and Nike might be justifiably proud of the quality of its laces. But if your generic laces snap (and so does your ankle), the problem is that the shoelaces you chose were defective. The problem is not that you chose someone else's laces.
Your stuff is yours. You can put anyone's milk on your cereal, not just the milk that Kellogg's endorses. If the milk you choose is sour and you have to pour your cereal down the toilet, that's too bad, but it doesn't mean that Kellogg's should have the right to shame you for having the temerity to choose the milk you preferred, and it certainly doesn't militate for DMCA-style enforcement to coerce you into using Kellogg's official milk in the name of protecting you from food wastage.
I raise this because there's a live debate over this subject, which is well represented in the latest 404 Media podcast:
In the podcast, Koebler and his 404 co-founder Joseph Cox express misgivings about various possible uses of "slop interop." Koebler discusses the fact that John Deere sells the same physical tractor at many different price-points, depending on which software a farmer chooses, and raises the possibility that a farmer might vibe-code software that replicates the high-end functionality without paying for it. Koebler discusses two possible downsides to this: first, that the vibe-coded software might be defective and expose the farmer to risk; and second, a question about whether it's fair or just for a farmer to get those additional functions of their tractor without paying for them.
I think the first objection is entirely legitimate. The software you run on your expensive tractor should be secure, both for your own benefit and for the security of our food supply. That's actually a pretty good reason to ditch John Deere's own software, which is very insecure:
But I think the second question – whether it's legit to get this functionality without paying John Deere for it – is spurious. Sure, if John Deere provides you with that functionality, then they can send you the bill for it. But if you provide yourself with that functionality, using software you procured independently of John Deere, then John Deere has no right to get paid for it, and they certainly shouldn't have the right to stop you.
That tractor is yours. The fact that John Deere manufactured it is irrelevant. If you pay a contractor to build a house for you and later on pay a different contractor to add another bathroom to it, you don't owe the first contractor anything. It's your house (and if that bathroom has a leaky toilet, that's a matter between you and the second contractor, and not a reason to force you to only use your original contractor).
The law defines a narrow slice of functions that manufacturers get to control: those ideas that a patent examiner deems "novel, useful and nonobvious." It's legit (and good) for commercial operators to copy and improve the ideas of their rivals and competitors. It's why skilled tradespeople cluster together in the same geographic area.
Take Clerkenwell, the neighborhood in London where precision machining was invented by the jewelers and machinists who clustered there during the Longitude rewards. The reason all those competitors sought such close promixity to one another was because they wanted to copy the new techniques their rivals were developing (even if that meant that their rivals would be able to copy them):
If you're old enough to remember the advent of spreadsheets and the rollout of Visicalc, Lotus 1-2-3 and Excel, you'll know that every time a software vendor invented a new spreadsheet feature, all the other ones immediately copied it. This wasn't bad for spreadsheets: it was good for spreadsheets, just as copying your competitors was good for precision machining in Clerkenwell.
Everything from GNU/Linux (a feature-for-feature clone of AT&T Unix) to SAMBA (a plug-compatible clone of Microsoft's SMB) to iWork (file-compatible, feature compatible clones of MS Office's apps) came about through this "adversarial interoperability." What's more, the heyday of this kind of adversarial interop was also the heyday of disenshittification, the era in which any anti-feature installed by your existing vendor could be countered by a disenshittifying patch from a new one:
The question of whether vibe-coded software is any good is separate from the question of whether it's good to replace the software in your devices, even if you replace it with software that has features that the manufacturer would normally charge a premium for. You are under no obligation to arrange your commercial affairs to the financial benefit of manufacturers, and anyone who claims to be surprised that someone they sold a device to used it to benefit themselves at the manufacturer's expenses is either disingenuous…or unserious. You bought it, it's yours.
But just as this vibe-coded adversarial interop has drawn both valid and spurious criticism, it's also drawn excitement that is partially justified and partially overblown.
When it comes to reviving bricked devices or rooting out the surveillance and ads from your smart TV, vibe-coded interop tools are genuinely exciting. These are the kinds of enshittification that can truly be remediated with individual jailbreaks.
But the real, meaty examples of enshittification can't be addressed on an individual level – they require systemic fixes. If you jailbreak your iPhone so that it can use alternative app stores, that fact is not sufficient to cause those alternative app stores to come into existence. If we want to stop Apple from levying a $100b "app tax" on the digital economy, it's not enough for individuals to vibe-code their own iPhone patches. We need to repeal the law that bans jailbreaking iPhones, so that a would-be alternative app store can raise capital and pay programmers and tech support staff and salespeople to assemble rival catalogs of software and help customers figure out how to get up their phones to use it.
The same goes for jailbreaking your printer so it accepts generic ink. Sure, this may work for you if you're up for refilling your own cartridges, and I'm sure that you can find someone on Alibaba who will sell you ink by the gallon. But the mere fact that a few users have secretly and privately jailbroken their inkjets won't conjure up a generic inkjet cartridge manufacturing sector. For that to happen, we need to make jailbreaking printers legal, not just possible.
Indeed, if we want to improve the quality of all that jailbreaking code the chatbots are writing and circulating, we need to be able to put those jailbreaks in public repositories for public inspection and public critique. It's possible for some of the code produced by an LLM to be slop, and for other code to be fine, and for yet a third tranche of code to be remediable slop that can be made robust and reliable through auditing and refactoring. You can't figure out which is which if you have to keep that vibe code a secret on pain of a long prison-sentence.
I think there's good cause to be excited at the prospect of getting your devices working better, or just working at all. I think that even if the code stops working after a while, that's still fine – either you got more use out of your device than you would have without the new code, or maybe you just make more code that runs for a while for you, and then you do it again.
These individual improvements to your devices and technology are worth celebrating. But they are no substitute for systemic intervention. If we're going to have a new, good internet, a post-American internet that truly disenshittifies America's defective tech exports and replaces them with robust, transparent digital public goods, we need to make reverse-engineering legal again. Once we fix our policy, then we can fix our code:
"Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com.
"Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com
“Once Is Enemy Action,” a science fiction novel about the origins of modern technofascism. Today's words: 533 (27469 total).
"The Post-American Internet," a sequel to "Enshittification," about the better world the rest of us get to have now that Trump has torched America. Fourth draft completed. Submitted to editor.
A Little Brother short story about DIY insulin PLANNING
This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.
Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.
"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla
READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
Login
