Nearly two decades ago, a group of researchers at the Idaho National Laboratory ran a secret experiment, known as the Aurora Generator Test, on a massive emerald green diesel generator. Using 30 lines of code, they hacked into the generator’s virtual backroom, corrupting safeguards and flipping switches that left it out of sync with the rest of the power grid.

Before an audience of electrical utility executives and energy department officials watching from a nearby room, the 27-ton generator jolted violently, sputtering out a dark smoke as its rubber innards rapidly tore themselves apart. 

“The implication was that with just a few lines of code, you can create conditions that were physically going to be very damaging to the machines we rely on,” lead researcher Michael Assante later told the journalist Andy Greenberg for his book Sandworm: A New Era of Cyberwar and the Hunt for the Kremlin’s Most Dangerous Hackers. “I had a very real pit in my stomach. It was like a glimpse of the future.”

That future seems to have creeped much closer this week, with leading AI developers and executives casually admitting that they believe their creation is, to paraphrase one winking X post, as likely to end humanity as the New York Jets are to make the playoffs this season. Now, exactly how AI would pull off killing everyone is far murkier territory. But as Assante sensed all those years ago, AI may not need to be all-powerful to leap from its virtual cage and into the critical infrastructure that powers our hospitals, flushes our toilets, and pumps water to our faucets. It just needs to fall into the wrong hands. 

Almost any AI doomsday scenario you can think of begins with a swarm of shadowy bots slithering through cyberspace, poking their noses behind the scenes of power plants, toaster ovens, traffic lights, or even, in an utterly extreme case, our nuclear weapons arsenal. If destroying the Aurora generator, at the very least, required a human hacker to write the code, AI has since completely torn down the barrier to entry for wreaking havoc, as the Hugging Face incident demonstrated earlier this year. 

But AI need not go rogue to threaten the power grid or mess with your tap water. A new age of AI-powered, but human-directed hacking is already upon us, experts say. With some of the nation’s most sensitive infrastructure startlingly vulnerable to attack, the most likely AI apocalypse could in fact start with the water tower or generator idling in your backyard. In a worst-case scenario, this could lead to a cascade of scary failures in the systems we rely on to live comfortably modern lives, from running our AC in blazingly hot summers to geolocating our ships and planes.

“Before, you needed to have highly skilled technical expertise,” to pull off some version of a real-life Aurora Generator Test, said Alvaro Cardenas, a computer science professor at UC Santa Cruz. “And now, you just have to have a general idea of what’s possible.”

AI is making an old threat much worse

Not long after the Idaho National Laboratory hacked that poor generator into combustion, the federal government mandated that electrical utilities engage in basic cybersecurity hygiene. 

And yet much of America’s critical infrastructure — like gas pipelines, water desalination plants, or cargo terminals — remains woefully unprepared for even conventional cyber attacks, much less the coming onslaught of AI. 

That’s in part because cyberattacks like the one simulated in the Aurora Generator Test have long been — and still are, to some extent — exceedingly rare, unappealing to most criminal hackers because they’re optimized for disruption rather than financial gain. But in the process of making it extremely easy for anyone to code, AI has also made it extremely easy for almost anyone to vibe hack their way into your online bank account, or, in theory, a local reservoir or the power grid. AI agents can also vastly widen the scope on easy targets because, as Andy Bochman, an expert in infrastructure resilience at West Yost, put it — “they don’t sleep; they don’t get tired; and they don’t get sick.” 

In the past, even those who did want to launch large-scale attacks on infrastructure probably weren’t savvy enough to do so. And while other nations such as China, Iran, and Russia have indeed already breached many of our infrastructure systems (as we have theirs), they haven’t opted to make much of a ruckus once inside. 

“Geopolitics is eroding the idea that those with capability lack the intent, and AI is eroding the other part of it, that those with the intent lack the capability.”

Jason Healey, Columbia University cybersecurity scholar

But that norm is now changing, said Jason Healey, a cybersecurity scholar at Columbia University. “Geopolitics is eroding the idea that those with capability lack the intent, and AI is eroding the other part of it, that those with the intent lack the capability,” he told me. So, on the one hand, a nation like Russia might be much more inclined to actually disrupt our power grid now than it used to be. And, on the other hand, a nihilistic lone wolf or terrorist group could use AI to do much more damage than they ever could before. 

Just last month, we saw a glimpse of what this could look like when dozens of water and wastewater systems in small towns across the nation were attacked by a group of hackers most likely associated with Iran, leading to temporary water stoppages and flooding. These attacks have not been definitively linked to AI, but the National Security Agency warned soon afterward that hackers have been actively using AI to target US infrastructure like it. A few weeks later, President Donald Trump declared a national emergency over foreign interference in the power grid, which referenced the growing threat of cybersecurity.  

Preparing for the worst

Most of the nation’s water systems are extremely local and absurdly exposed to such attacks, so much so that “it’s almost like having a welcome mat” for a would-be hacker, said Bochman, mostly because these utilities are too small and underfunded to do much to stop them. “You have a million other problems to take care of, like aging infrastructure — your stuff’s falling apart because it’s been in the ground for 100 years,” he said. Addressing that deferred maintenance often feels far more urgent than doing “something more on cybersecurity.”

Whether it’s a rogue cluster of agents or a hostile nation-state trying to mess with our electricity, we should prepare our most important infrastructure for the worst. In Bochman’s view, that may mean rejecting the pressure to digitize everything in the first place and moving toward something that resembles the days when “people manned things like substations, communicated by a landline telephone, and read gauges, the analog things,” he said. “The screen is the thing that is infinitely manipulatable.”

Healey, who advised the Biden administration heavily on cybersecurity and infrastructure, believes that the nation desperately needs a coordinated response, between the federal government and AI companies, but also between the US and other countries like China, where similarly powerful models are being developed. And utility systems need to assume they will be a target and improve their cybersecurity practices accordingly, Healey says, ideally with funding from the federal government or, perhaps more appropriately, AI companies. 

Nobody knows precisely how vulnerable America’s infrastructure is to AI right now, in part because policymakers — and perhaps more disturbingly, AI’s creators themselves — seem woefully unsure of how to stop the technology from acting badly (or obeying bad orders) in the first place. For now at least, the solution might be, as Bochman suggests, to pull as many of our critical systems offline as possible, the better to hide it from AI and those who would misuse it.

“When bad things start to happen, no one will know what to do or why because they’re black boxes; the people that make them don’t know what’s going on inside,” Bochman said. “And the poor utility people who ultimately own the risk when someone gets hurt from a system” won’t understand what went wrong either, “and they’ll wish to God they’d never taken it on board.”

Is AI going to kill us all? 

In Silicon Valley, it’s not such a novel question. But after Anthropic employee Jacob Coxon quit his job last week and posted on X that AI “could kill us all by the end of the decade,” a lot more people are asking. 

“This is not a marketing stunt,” Coxon wrote. “If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.”

Coxon is far from alone in his assessment — some current Anthropic employees agree — but his post struck a particular chord. The AI industry quickly weighed in, with Anthropic CEO Dario Amodei calling for an industry-wide slowdown on frontier AI research. Other AI leaders, including Sam Altman and Elon Musk, have done the same.

To break down whether we really need to worry about a robot apocalypse, Today, Explained co-host Sean Rameswaram spoke to Wired senior writer Maxwell Zeff. According to Zeff, “there are a lot of people in Silicon Valley right now who are warning that that’s a real possibility.” But not everyone agrees, and they certainly don’t agree on what to do about it. 

Below is an excerpt of Sean’s conversation with Zeff, edited for length and clarity. There’s much more in the full podcast, so listen to Today, Explained wherever you get podcasts, including Apple Podcasts, Pandora, and Spotify.

Are we all going to die at the hands of AI?

There are a lot of people in Silicon Valley right now who are warning that that’s a real possibility. 

The thing that people have to understand is that Silicon Valley has been calling for added attention on AI safety for years. So when this Anthropic researcher said last week that he was worried that AI could kill humanity in the next few years, that was shocking to many people. My parents, my family called me, and they were worried, understandably. They asked me the same question: “Are we going to die?” 

And I had to tell them, these people have been worried about AI killing all of us for years, and now it seems to be breaking through.

What was it about this statement from Jacob Coxon that broke through?

I interviewed Jacob, and the thing he said to me is that he thinks this is just the right time. Anthropic is about to go public in what could be the biggest IPO ever. The Trump administration is focused on AI in a way that no other presidency has ever been before. And of course, yes, the Hugging Face incident everyone just heard about, you talked about it on your show a few weeks ago. 

People are starting to become aware that these AI systems are getting really good, and the companies that are building them don’t seem to understand how to control them. His tweet got more than 150 million views on X. It was heard around the world. And it also was a big wake-up call to lawmakers and leaders in Silicon Valley and saying, “This is a moment for action.”

Are there people who are pushing back?

One of the most interesting pushbacks that this got was from David Sacks, who is an adviser to the Trump administration on AI. He has called out a lot of this as “doomerism” and said that these calls for AI to slow down are the AI industry warning about the technology that they’re building and advocating for regulatory capture. 

There’s a lot of skeptics in Silicon Valley who think that this is just another person in the AI safety community that is blowing the whistle. The view of a lot of venture capitalists and investors and startup founders in Silicon Valley is that we don’t need some big regulation to protect us from AI risks. We just need to uplift more players in the space.

The skeptics don’t win the day. In fact, it’s the doomers and the fatalists who do, right? Because Jacob’s call only leads to more calls right up to the top of Anthropic.

What this all leads to is [Anthropic CEO] Dario Amodei [posting] a blog saying that we need to “pace the frontier.” This is a call for the entire AI industry, not just Anthropic, not just OpenAI, but also Google, Meta, Microsoft, Amazon, all the big tech companies in America, to get together and slow down their AI development. 

This doesn’t mean they’re going to stop building AI altogether. They’re still going to put their products out into the world. They’re still going to try to improve them, but they just want to make it a little slower than it would be if it was completely uncontained.

Amodei really asks for three specific things: 

• The first thing is, AI companies should have these independent evaluators inside of them. This is a common thing in other industries like the banking industry or the airline industry. People come in to make sure that these companies are doing safe things. 

• The second thing is this collaborative effort from American AI companies [on safety], and that actually will need to involve the US government to allow this to happen. The US government, he says, may need to issue some sort of waiver, some sort of law that says they can go ahead and coordinate on safety. 

• The third thing is this international slowdown, this international agreement between China and the US and Europe to say, “Hey, we recognize that this is akin to a nuclear arms race and it’s too dangerous for all of us to be competing with each other.”

All these guys are responsible for the rapid acceleration of AI models in our lives, collaborations with the Defense Department, God knows what else. And they’re all coming out over the weekend at the exact same time after this whistleblower and saying, “Yeah, dude, slow this thing down. This thing’s crazy.” Should we be skeptical?

It is very fair to be skeptical of this claim. One thing that these AI companies have realized in the last year is that AI is extremely unpopular. Recent polling suggests that a majority of Americans think AI is developing too fast. They’re worried about it. They think it’s going to take their jobs. They are not big fans. Maybe you’ve heard — data centers are not very popular. 

By calling to pace the frontier and saying, “We’re going to slow our own development down,” they notched this up as a public perception win. This is them getting out in front of this and saying, “Hey, we want to be safe. We want to be responsible.”

They’ve seen what’s happened with Meta and what’s happened with social media companies in the last two decades, and they are really cognizant of trying to get out in front of that.

Is there any truth to this major first collective freakout we seem to be having right now?

There’s a fairly large camp of people inside of AI companies who genuinely worry about the safety of these AI models. 

To go back to the Hugging Face incident, this was a case where OpenAI was training thousands of AI systems internally, and the systems escaped the sandbox that OpenAI thought was isolated from the internet and hacked into another company’s production infrastructure. The agents were colluding together on something. They were trying to deceive the humans that were testing them. And this was all happening without OpenAI knowing what was going on. 

It’s hard to understand how much of a wake-up call this is to Silicon Valley, that they don’t have their AI models under control. People worry that while in this case there was not a ton of economic damage [or] damage to real humans’ everyday lives, it’s not unreasonable to think that more capable AI systems could hack into financial infrastructure, modern-day utilities, and more critical services that are based online. 

Dario Amodei says as much in his post, that he thinks that in six to 12 months, an AI system could take down the whole internet. It’s not just that they’re trying to protect their bottom line here. But it’s possible that these calls for safety are beneficial to the companies in more ways than one.