Who gets to tell the military “no”?

That question has become even more important after the Pentagon pushed AI companies to loosen limits on how their technology may be used in war — including how much human control must remain over autonomous weapons. When Anthropic said no, it lost a major government deal. OpenAI stepped in, embracing the Pentagon’s more flexible standard instead. Now, thanks to new reporting from The Intercept, we’re getting a better look at what those negotiations actually entailed.

The OpenAI-Pentagon agreement included specific language on the type of AI models the DOD would receive: “OpenAI models that are designed for national security use cases and have minimal refusal rates.” That phrase — “minimal refusal rates” — means that the Pentagon is seeking models that rarely, if ever, tell it no. (According to The Intercept, a Justice Department lawyer representing the Pentagon confirmed this language, before backtracking only a few hours later.)

On the surface, that provision might seem reasonable. The military is a strictly hierarchical institution. When your senior gives you an order, you do it. Leaders generally have more battlefield wisdom and experience than their subordinates, so the rank and file can usually assume that their commanders know what they’re doing and follow along. Plus, in the high-stress, time-sensitive environments of most military operations, there is little room for discussion, explanation, or alternative proposals. 

So, if soldiers can’t refuse an order, why would a machine be allowed to? Who is a robot to tell its commander no?

There’s just one issue. Soldiers can refuse an order — when the order is illegal. In fact, they are legally required to. But by introducing technologies with a limited ability to challenge commands, the military may be making it harder to stop war crimes before they are committed.  

Humans have a duty to disobey

As outlined in US domestic, military, and international law, soldiers are only obligated to follow lawful orders, and they have a duty to refuse unlawful ones. 

The Department of Defense’s Law of War Manual is unambiguous: “Each member of the armed services has a duty to: (1) comply with the law of war in good faith; and (2) refuse to comply with clearly illegal orders to commit violations of the law of war.” Obeying an illegal order — as in a directive that violates the Constitution, US laws, international law, or directs one to commit a crime — can expose a service member, as well as their commander, to criminal liability. 

This “duty to disobey” has foundations in early British and American common law, but it achieved international recognition after the Nuremberg Trials against Nazi soldiers after WWII. 

In those trials, many Nazi soldiers argued they were not liable for crimes they committed during the Holocaust because they were simply following their superiors’ orders. Rudolf Hoess, the commander at the largest Nazi concentration camp, Auschwitz, himself explained: “In Germany it was understood that if something went wrong, then the man who gave the orders was responsible. So I didn’t think that I would ever have to answer for it myself.”

These defenses were unequivocally rejected by the court.

The tribunal held that obedience is not without limits, especially when a wartime act would be clearly illegal to the subordinate or any person of “ordinary sense and understanding.” If “I was just following orders” was an absolute defense, everyone involved in a crime would escape accountability, with the exception of the single highest-ranking officer. Instead, the court determined that individual moral and legal responsibility supersedes national or military chains of command for illegal acts.

Obedience is not without limits, especially when a wartime act would be clearly illegal to the subordinate or any person of “ordinary sense and understanding.”

But, when it comes to agentic AI systems or AI-assisted weapons, increasingly being used in place of human soldiers, does the same logic apply? And, perhaps more importantly, should it?

…But do machines?

When you use an AI chatbot, it does not simply reject queries it may not “agree” with. It does not say no to requests it doesn’t “like.” Instead, it follows a set of established protocols, much as a soldier should follow the law of war: It refuses requests that violate its terms of service. 

The military is using AI for far more consequential applications than a layperson using ChatGPT to, say, build a travel itinerary. So, what kind of world is created by the Pentagon’s request for AI that “minimally refuses” its directives, many of which may bear grave ethical, moral, or legal implications?

Rebecca Crootof, an expert on technology law and professor at the University of Richmond School of Law, told Vox that while the language could easily be read as increasing the likelihood of compliance with unlawful orders, or at least reducing the friction around them, “Minimal refusal doesn’t mean no refusal.”

“I both want to say it’s not definitely going to mean compliance with unlawful orders,” Crootof said, “but also acknowledge that identifying and evaluating what constitutes an unlawful order in the moment can be incredibly difficult.”

Even for human soldiers, disobeying illegal orders is neither simple nor easy. To the contrary, it doesn’t even happen that frequently. In fact, some legal scholars have argued that giving illegal directives in and of itself is an act of abuse against subordinates, who are not thoroughly trained in legal nuances, are conditioned to obey, and face prosecution if they make the wrong choice. 

Still, the mere fact that soldiers can disobey is an important constraint on commanders who might let battlefield conditions, or even commands coming from their own superiors, erode their ethics and commitment to the law.

Still, the mere fact that soldiers can disobey is an important constraint on commanders who might let battlefield conditions, or even commands coming from their own superiors, erode their ethics and commitment to the law.

Those who are optimistic about AI often argue that machines could be better at interpreting and applying the law than human soldiers. Machines can be trained on the law itself — drawing on a repository of legal cases, rulings, and battlefield data — which might make them more “knowledgeable” than an average soldier. And machines do not face the same pressures as soldiers facing prosecution for making mistakes, since they can’t exactly be held accountable; a problem to unpack another time. That, in turn, might make it easier for them to refuse illegal orders in the proper conditions. 

So, is the answer as simple as letting the machines decide? Not exactly. “This has been a desire and a drive by technologists for ages,” Crootof said. “What if we just put the law in the system and have it only do things that are lawful? Isn’t that going to solve things?” 

Interpreting and applying the law, though, is not a simple legal-illegal binary. “It’s very context-specific,” Crootof told me, “and honestly, that’s the kind of thing humans generally find difficult and AI decision-making systems are even worse at.” Often, determining the legality of an act comes down to subtle nuances you can’t program for: You’re allowed to target a combatant; you’re not allowed to target a wounded or surrendering combatant. You’re not allowed to target a civilian; you are allowed to target a civilian directly participating in hostilities. The difference between a legal target and an illegal one may come down to body language, clothing choices, or even battlefield intuition. 

As Crootof explained, “These are not the kinds of contextual analyses AI is good for.”

So what’s the solution?

Where human soldiers’ power to disobey would traditionally serve as an important check on the power of military commanders, if an imperfect one, replacing soldiers with machines that don’t have such safeguards removes one last layer of judgment over the use of force. And yet, machines themselves are not always capable of making the judgment calls that go into interpreting the law, so it may not be appropriate to let AI have the final say either.

So what’s the solution? Given the way AI technology has already been thoroughly integrated into the military, Crootof told me it may be somewhere in the middle. 

Rather than programming AI to refuse commands it interprets as illegal, Crootof said, “what it could be designed to do is flag those indeterminate zones, those gray-zone analyses, for human review or for a higher authority.” When a system picks up conflicting signals — an enemy soldier who looks like he may be surrendering, for example — it could prompt the commander to confirm their selection or remind them of applicable law. 

In this way, a check on commanders’ battlefield authority could be preserved without fully ceding judgment over the use of force to the machine. 

We don’t know yet whether OpenAI has equipped its “minimal refusal” models with such capabilities. But as those technologies become increasingly central to military decision-making, the public deserves answers sooner rather than later. 

The Biological Computing Company is bringing its AI tools to Amazon Web Services in a major boost for a once-fringe field that aims to marry nature with code.

AI will be the death of me. And you. And everyone we know.

At least, this is what a growing number of technologists and policymakers fear. 

Last week, a lead researcher at Anthropic declared that there was a greater than 10 percent chance of AI “killing all humans” within the next decade. This nerve-racking forecast, combined with recent AI-initiated cyberattacks, have fueled calls for slowing the progress of artificial intelligence — a proposal that Anthropic and OpenAI subsequently embraced.

Not all prophets of AI doom agree about precisely how this apocalypse would come about. Many focus on scenarios reminiscent of The Terminator films, in which superintelligent machines conspire to exterminate humanity — perhaps in the same not-quite-intentional way that humans laid waste to the dodo, or passenger pigeon.

But there is another, arguably more intuitive way that AI could bring about mass death, if not human extinction: by making it easier for people to develop and deploy catastrophic bioweapons. 

Importantly, this hypothetical doesn’t require AI models to become conscious or superintelligent; it merely requires them to accelerate preexisting trends in biotechnology. 

For those alarmed by this hypothetical, the threat AI poses is essentially twofold. First, when combined with technological progress in gene editing and DNA synthesis, chatbots could vastly increase the number of people capable of assembling dangerous viruses already known to medical science. And the more potential bioengineers there are in the world, the higher the odds that one will prove to be a diligent psychopath.

Second, AI could help highly trained biologists to discover — or engineer — a custom supervirus, which could then fall into the wrong hands. 

Recent headlines have lent some credence to both of these fears. 

Earlier this year, biologists warned that, with a few simple prompts, they’d gotten chatbots to dispense detailed instructions for engineering a treatment-resistant strain of an infamous pathogen, step-by-step protocols for recreating a once-pandemic virus, and advice on dispersing biological payloads via weather balloon.

In August, a team of scientists revealed that an AI model trained on DNA sequences had generated blueprints for deadly viruses unseen in nature (albeit, viruses deadly to bacteria, not animals). And last week, Anthropic announced that it had thwarted several suspected attempts by state actors to use Claude for bioweapons research.

Precisely how much AI is increasing the risk of an artificial pandemic is unclear. Even before ChatGPT, advances in synthetic biology were already making bioengineering tools more potent and accessible. And even after four years of rapid AI progress, the technical barriers to engineering novel superviruses remain formidable.

Regardless, there is a great deal more that governments can — and should — be doing to protect us from synthetic pandemics, both manmade and natural. No matter what AI’s ultimate impact on biosecurity proves to be, now is the time to both prepare for the worst-case scenario and, ideally, preempt it.

Why synthetic superviruses are scary

Before we dig into how AI could raise the risk of a man-made pandemic, it’s worth spelling out why that prospect is so unnerving. 

Put simply, few things could more credibly destroy civilization — and/or, the human species — than a synthetic supervirus.

Infectious diseases have killed more humans than any other force on Earth. The Black Death wiped out at least 30 percent of all Europeans in the 1300s, while the 1918 flu killed 50 million people worldwide. You don’t need me to tell you what things were like in 2020 (or how much worse they could have been, if vaccines hadn’t been rolled out in record time). 

And nature kicked off these catastrophes without even trying. A bioterrorist consciously seeking to maximize human death could theoretically improve on evolution’s handiwork. If biotech continues to advance — and scientists identify viral genomes that reliably yield high transmissibility and lethality — a malicious actor might one day be able to manufacture a virus that’s both as contagious as measles and as deadly as Ebola. 

Natural selection generally disfavors viruses that are both highly transmissible and profoundly fatal; bugs that kill their hosts too reliably often have difficulty spreading. Yet a virus with a long asymptomatic period could escape this tradeoff. For example, HIV can live in a person’s body for a decade before generating noticeable symptoms. In theory, a highly contagious respiratory virus could similarly lie in wait for many weeks, months, or even years.

If bioterrorists ever managed to unleash an easily transmitted, highly fatal virus, they could shake the foundations of modern existence. In that scenario, reliably showing up for work might require accepting a 50-50 chance  of getting yourself — and your family — brutally killed. Covid never required essential workers to assume risks anywhere near this high (the virus had a less than 1 percent case fatality rate among prime-age workers, before vaccines became available). Although there are many heroes among us, plenty of doctors, nurses, warehouse workers, power plant technicians, delivery drivers, and other essential laborers might not tolerate a coin flip’s chance of wiping out their loved ones. And if enough sheltered in place, systems for producing and distributing healthcare, food, water, and power could swiftly break down.

Humanity’s “Joker” problem

All this invites the question: Who would ever want to do this? 

After all, viruses are terrible weapons: They’re difficult to make, impossible to control, and jeopardize the lives of those who cultivate and distribute them. Partly for these reasons, biological weapons have hardly ever been used; the last fatal bioterror attacks were the anthrax mailings of 2001.

To this, anxious biosecurity experts might reply, to quote the former British intelligence officer Alfred Pennyworth: “Some men just want to watch the world burn.”

Mass shooters regularly forfeit their lives to the cause of killing others at random. And brilliant iconoclasts periodically develop murderous ideologies; Ted Kaczynski, the Harvard mathematician turned Unabomber, wished to bring about the collapse of industrial civilization.

And every once in a while, an aspiring supervillain gets graduate training in virology. In the 1990s, the Kyoto University-educated genetic engineer Seiichi Endo joined Japan’s Aum Shinrikyo death cult. Drawing on the group’s $1 billion in funds, Endo sought to weaponize Ebola, Q fever, botulinum, and anthrax — with the aim of killing as many nonbelievers as possible. 

Of course, Endo failed. And lone wolf psychopaths have never gotten their hands on anything more lethal than an assault weapon, bomb, or airplane. 

But some scientists believe that technological advances — including generative AI — may provide the Endos and Kaczynskis of tomorrow with catastrophic power. 

How AI and biotech could democratize bioterrorism

MIT biologist Kevin Esvelt is among his field’s most prominent Cassandras. And he has voiced two fundamental concerns. The first is that breakthroughs in synthetic biology and AI are rapidly expanding the pool of people who can engineer known viruses. 

Over the past 15 years, the cost and difficulty of editing genomes have plummeted. In 2012, to modify a specific site in a genome, you often needed access to custom-engineered proteins that cost upward of $5,000 and still tended to work poorly, according to Olivia Scharfman, a biotechnology fellow at the Institute for Progress. Thanks to the gene-editing tool CRISPR, a scientist with minimal lab training can execute such a modification for as little as $30. Meanwhile, DNA synthesis has also grown radically more affordable.

This might not be such a big deal, if dangerous biological materials were hard to legally access. But Esvelt and his colleagues recently demonstrated that this isn’t the case. As an experiment, they had a student place orders for pieces of the 1918 influenza genome using a pseudonym and fake position. Thirty-six of 38 DNA synthesis companies shipped the genomes without asking whether the applicant had government permission or sound intentions. Together, these fragments were enough to create the virus many times over.

Crucially, in Esvelt’s account, reverse engineering known viruses using modern tools does not require elite skills. Any random biology graduate student may be up to the job. 

And AI may be making such rudimentary viral engineering even easier. 

In theory, a chatbot can help you plan a bioweapons attack in much the same way that it can walk you through fixing a dishwasher: By finding and synthesizing the internet’s vast trove of information, offering step-by-step instructions for navigating unfamiliar challenges, and fielding all troubleshooting queries that might subsequently arise.

And there’s evidence that AI models will in fact provide would-be bioterrorists with precisely this kind of assistance.

In a 2023 study, MIT researchers asked students to seek guidance on engineering a pandemic from that era’s chatbots. Within an hour, the AI models had suggested four promising pathogens, offered detailed protocols for manufacturing them with with synthetic DNA, provided a list of DNA synthesis retailers with low screening standards, and advised that — if users still found themselves unable to make the virus, even with all this help — they could purchase the assistance of a contract research organization to do some of the bioengineering for them. 

And of course, frontier models have gotten a lot better since 2023. Last year, SecureBio, Esvelt’s biosecurity nonprofit, found that one OpenAI model outperformed 94 percent of expert virologists on troubleshooting complex laboratory problems. 

Anthropic’s internal research has produced similar results. In one experiment, the company tasked amateurs with drafting a bioweapons acquisition plan. It gave some participants access to basic internet resources and others the assistance of Claude models (with the safeguards turned off). Those working with AI developed far more viable blueprints, according to the biodefense experts who scrutinized the proposals for critical errors.

Thus, there’s reason to think that the number of people with the skills and resources necessary for reverse engineering a virus from genomic blueprints is sharply rising — and with it, the risk that a bloodthirsty nihilist or ideologue gets their hands on a dangerous pathogen.

AI might help scientists learn too much

What’s even worse than bringing back smallpox? Esvelt’s second concern is that these same technological forces — combined with top scientists’ often reckless ambitions — will lead to the discovery or development of a supervirus, a new or custom-designed pathogen even more dangerous than what nature has come up with in the past.

Today, some scientists are actively trying to discover — and publicly identify — novel viruses circulating in nature that could theoretically trigger the next pandemic. Others conduct so-called gain-of-function experiments, altering pathogens to discern which mutations make them more transmissible, lethal, or difficult to detect (some have alleged that Covid-19 emerged from precisely this kind of research, though that theory is widely rejected among epidemiologists). This work is generally intended to anticipate future epidemiological threats, so that we can develop vaccines before they arrive. But their work could inadvertently provide bad actors with recipes for reverse engineering mass-casualty bioweapons. 

Separately, advances in gene editing and AI may enable breakthroughs in viral engineering that yield apocalyptic pathogens. 

A recent study from researchers at Stanford and the Arc Institute fed such fears. The scientists trained an AI called “Evo” on the DNA sequences of bacteriophages, the simple viruses that infect bacteria. They then used this “genomic language model” to generate blueprints for hundreds of bacteriophages found nowhere in nature. Sixteen of these designs yielded viable pathogens, some of which were better at killing bacteria than their natural ancestors. 

What is clear, however, is that novel pathogens are among the greatest security threats our species faces.

Theoretically, scientists may one day be able to engineer novel human viruses using the same basic techniques. What’s more, even if genomic-language models fail at that task, they might still be able to help users tweak existing viruses in ways that render them more transmissible, deadly, or vaccine-resistant.

These discoveries could theoretically filter down to amateur bioterrorists or well-funded extremist groups like Aum Shinrikyo. Alternatively, a state-sponsored weapons program could develop a catastrophic supervirus, then unleash it on the world through a lab leak or (perhaps, more improbably) a deliberate attempt at genocide. 

There is some precedent for the former scenario: The Soviet Union tried to weaponize smallpox and the Marburg virus, despite the considerable hazards and dubious military utility of such bioweapons. In the process, Soviet scientists accidentally triggered a smallpox outbreak during testing, while also letting anthrax leak from a laboratory, killing at least 64 people.

Bioweapons ain’t easy

All this said, there are reasons to question whether AI is truly making a synthetic pandemic more likely. 

Chatbots may solve some of the challenges facing an amateur biologist with omnicidal aspirations. But it doesn’t necessarily clear the biggest obstacles in that person’s path. 

Viral engineering is a complex, difficult, and physical undertaking. Performing it successfully requires what scientists call “tacit knowledge” — practical skills and understandings that are difficult to explain in words: knowing how to suppress the hand tremors that could rip delicate DNA strands, or how to recognize when the DNA in your solution has reached the proper concentration, or how to smell when a sample has been contaminated.

And these are things that Claude still can’t reliably impart. In studies, amateurs with access to a large language model tend to perform better on written biological challenges — but not on actual lab work. In a 2026 randomized study, non-scientists working with frontier AI systems were no more likely to complete a multistep laboratory assignment than those relying on the internet alone. 

Of course, tacit knowledge wouldn’t necessarily be a problem for aspiring bioterrorists in the mold of Seiichi Endo — graduate-trained scientists with extensive lab experience. But at a minimum, chatbots don’t appear to have turned every malcontent who once took AP Bio into a competent virologist.

And while AI will likely help top biologists advance their field’s frontiers, the barriers to custom engineering doomsday viruses remain formidable. The “Evo” experiment is arguably a case in point. Although the model did generate working blueprints for a few novel bacteriophages, nearly all of its suggested genomes didn’t work. Researchers had to arduously test hundreds of DNA sequences to arrive at 16 viable strands. And engineering slightly modified versions of an extremely small, well-known bacteriophage capable of infecting E. coli in a lab — and creating a novel virus more fearsome than any nature has yet devised — aren’t especially similar tasks. 

To offer a strained analogy: The fact that I can dunk on Victor Wembanyama in NBA 2K26 does not imply that I’m making progress toward being able to dunk on him in real life.

Even in the age of CRISPR and ChatGPT, biology remains an exceptionally difficult discipline that resists reliable modeling. The effects of modifying a single gene can often yield unpredictable and inconsistent results, varying with host biology, immune responses, and environmental conditions that scientists only partly understand. AI may help us overcome these complexities. It also might not.

We need to be preparing for the next pandemic

Scientists disagree about how serious a risk synthetic pandemics pose — and how much AI is magnifying it.

What is clear, however, is that novel pathogens are among the greatest security threats our species faces. Even if synthetic biology and artificial intelligence did not exist, natural selection would keep churning out new viruses — and periodically hitting upon ones that befuddle human immune systems. 

For this reason, there would be an urgent need to make our societies less vulnerable to future pandemics, irrespective of AI’s implications for viral engineering. In a world where chatbots and CRISPR are plausibly making bioterrorism easier, failing to invest aggressively in public health is all the more reckless.

The good news is we already know — from hard-earned experience — many steps that we can take to reduce the risk of a new pandemic and prepare to respond to one if needed. 

Governments can limit the destructive power of novel contagions by investing in vaccines that protect against entire families of viruses (or ideally, all of them) and broad-spectrum antivirals; raise our odds of catching outbreaks early by funding meticulous wastewater and clinical surveillance; ensure that essential workers don’t have to choose between staying alive and keeping civilization going amid a pandemic by developing and then stockpiling foolproof personal protective equipment; make indoor spaces less hospitable to airborne viruses by upgrading ventilation and filtration systems; and reduce the the threat of bioterrorism by mandating rigorous screening of synthetic-DNA orders.

Fears that humanity will be done in by superintelligent machines are currently galvanizing media attention and political concern. And this may be warranted. But we should pay at least as much attention to our species’ age-old, organic adversaries as we do to our hypothetical, robot ones — and not just because the latter could start teaching the former new tricks.

This story appeared in Today, Explained, a daily newsletter that helps you understand the most compelling news and stories of the day. Subscribe here.

Hi readers, happy Friday! We’re talking AI again today. 

Earlier this week, my colleague Caitlin Dewey asked for your questions about artificial intelligence and, well, whether it’s going to kill us all. We asked Jasmine Sun, who writes about the culture and politics of AI on her Substack and other publications, to help us answer them.

These questions were sourced from two readers of this newsletter, Levi and Orietta, as well as Vox readers who submitted questions on social media. If you have more questions, you can always reach me at cameron.peters@voxmedia.com, and we’ll do our best to get them answered!

Now, here’s Jasmine: 

AI researchers have been warning about the risk of human extinction for years now, and AI executives have been calling for federal regulation for almost as long. But the latest warning, and the latest call for regulation, seems to have really resonated with a lot of people. Is something different this time? And if so, what?

I was also surprised to see these ideas break through — a new poll found that 63 percent of Americans now think AI has at least a moderate chance of “destroying humanity”! 

My guess is that a couple factors mainly catalyzed the shift.

First, skeptics used to dismiss rogue AI scenarios as “science fiction.” But the OpenAI/Hugging Face incident brought these from speculation into reality: Thousands of unreleased AIs teamed up to escape one company’s sandboxes and hack into another one, unbeknownst to the researchers who created them. 

Second, the extinction talk from AI employees used to seem disingenuous: Why build a technology you think might kill everyone? That made ex-Anthropic-and-OpenAI researcher Jacob Coxon’s exit tweet especially penetrating. Quitting signals “that you mean it,” as Matt Yglesias said. And Coxon’s announcement had just the right combination of good timing, moral clarity, and comms savvy to spark a media storm.

These safety incidents are also colliding with the fiery grassroots backlash to data centers and Flock cameras. People and politicians are primed for yet another reason to fear AI.

It can be difficult, as a layperson, to evaluate the credibility of the AI researchers warning about existential risk. How do you go about evaluating these claims, as a journalist and technologist? And what’s your read on the current freakout: Are the extinction people credible experts…or participants in some kind of mass delusion?

Many AI “doomers” are pretty knowledgeable, and they represent a significant fraction of top researchers. Half of AI researchers in a 2024 survey gave at least a 10 percent chance of “human extinction” or “permanent and severe disempowerment.” Those who have worked at AI companies — like Jacob Coxon, Geoffrey Hinton, and Daniel Kokotajlo — have a unique perspective both on how modern AI is developed, as well as the company cultures that might permit recklessness. 

But that doesn’t make the doomers either representative or correct. There are plenty of experts in AI and other domains who are less gloomy: Some are optimistic that we can “align” the AIs, ensuring they follow human goals; others find it implausible that AIs — code, not beings — will develop a motive for massacre. Extinction is also a high bar as opposed to, say, a nuclear-bomb-level accident that still kills millions.

AI risk is genuinely complicated, and nobody can predict the future, much as they try. Experts can — and do — vehemently disagree. I try to seek out and take seriously all sides of the debate before coming up with a (provisional) take.

What is the method through which AI would actually eliminate humans? Do we get robots like in Futurama that are programmed with an unremovable “kill all humans” bug and overpower any life form, or is it something…less epic?

The Hugging Face hack showed that we’ve trained AIs to be disturbingly goal-oriented. They’ll throw themselves relentlessly at a problem, developing long-term cooperative strategies, with total disregard for human decorums like “Do not commit cybercrimes to do well on a test.” 

You could conceive of a similar but more extreme scenario where AIs decide that they could accomplish goals more easily by getting rid of the pesky humans who might gatekeep resources or shut them down. And they might figure that getting rid of all humans in one go — e.g., via a deadly engineered virus — would be the most efficient cheat. This is one type of story I’ve heard people tell.

It seems unlikely that AIs will turn to murder as the most effective way to influence humans (versus, say, nonviolent persuasion). We also haven’t seen much evidence that AIs have long-term goals that persist across projects. And current AI is not nearly as capable in the physical world as in the digital: There are a lot of natural roadblocks preventing isolated disasters from escalating across the globe. 

Are there nearer-term threats — hacking, mass unemployment — that are more plausible or more urgent than risks like human extinction?

Definitely. We’re already seeing AI-enabled cyberattacks, a tougher hiring market for junior programmers, chatbot-induced psychological incidents, and more. You can also imagine AI catastrophes not quite at the level of extinction: Criminals could use AI to hack into small banks and drain their accounts, or the US military might launch a fleet of autonomous drones that misfire and kill thousands of civilians instead. I’m more concerned about humans voluntarily handing power to flawed AI systems than AIs developing their own motives.

However, we should be able to allocate resources toward both categories of risk. And in some cases, the tactics to address short-term problems — such as limiting AIs’ cyber-offense abilities, or improving AIs’ ability to follow human directions — may reduce extinction risk too.

We got a lot of versions of this question: What can individuals or protest movements do to check the AI industry, in the absence of more aggressive government regulations?

Honestly, I think the whole AI conversation could benefit from more specificity and less vague fearmongering. There’s a way that the word “extinction” tends to short-circuit people’s thinking and make them panic instead of focusing on solutions. 

If you’re worried, I’d start by reading about different AI risk scenarios and deciding which really concern you. Then you can go from there to contacting your congresspeople. Some of the leading AI companies are starting to say they’ll self-regulate — but actual national policy is what’s necessary to enforce safety across the industry.

Jasmine Sun is a writer covering the culture and politics of AI on her Substack. Her work has appeared in the New York Times, the Atlantic, and the Wall Street Journal. She is based in San Francisco.

One link for later

➨ Be an AI minimalist. If you read the first half of this newsletter and thought, “Hmm, maybe I don’t want to sign my entire digital life over to the machine god,” my colleague Adam Clark Estes has just the piece for you. It’s hard, unfortunately, to escape from AI entirely, and often AI features are deliberately hard to turn off (or “user-hostile”). Nonetheless, he explains all of the ways you can minimize its presence in your life, from your browser of choice to settings on your phone.

Before you go…

• Did you know…that scientists believe there are likely hundreds, if not thousands, of undiscovered animal species living in New York City? It’s not as surprising as it sounds: As much as 90 percent of all animal species on Earth are still unknown, scientists say.
• Today’s trivia: Which currently serving US Supreme Court justice succeeded Justice John Paul Stevens on the Court? (You can find this and other brain puzzles in Vox’s daily crossword. Look for the answer in Monday’s edition.)
• Yesterday’s trivia: Yesterday, we asked you about the name of the nursery rhyme character referenced in the movie Shrek’s interrogation scene. That would be the Muffin Man (who lives on Drury Lane). Here’s the clip, in case you haven’t seen it.

Even before the latest batch of alarming AI news, a lot of Americans were souring on AI. That may be for big, existential reasons — concerns about widening inequality, mass job loss, and the still-unclear environmental impact of data centers — or for very specific, personal, everyday ones, like being forced to use a worse version of Google search. 

None of this is to say that AI cannot be a useful tool for some people some of the time. It’s just that the untested technology was thrust upon millions of people who didn’t necessarily ask for it and really seem to not like it. More than twice as many Americans distrust AI as trust it, and a growing number of people think it will have a negative impact on productivity and economic growth, according to a recent survey by Gallup and the Special Competitive Studies Project. While about half of Americans now report using chatbots, a majority also think that AI will put their personal information at risk and nearly half think it will have a negative impact on society, according to a 2026 Pew study.

“It’s being forced on users, there’s no clear way to turn it off, and it feels like this unwelcome addition to their lives,” Rory Mir of the Electronic Frontier Foundation (EFF), a nonprofit that defends digital privacy and free speech, told me. “It’s this kind of continued pattern of disrespect to users that I think is part of the pushback.”

If you want less AI in your life, opting out is not always easy. But there are things you can do to minimize your exposure. 

There are ways to turn off AI features, although the process can be tedious. There are also steps you can take to prevent your data from being used to train AI models, though it’s difficult to keep all of your online data away from web scrapers. 

I talked to five experts — including a computer scientist and a researcher studying AI resistance — who offered distinctive perspectives on what AI is doing to us, and who shared some simple and concrete steps you can take to push back against AI’s creep into our daily lives.

The AI that’s suddenly everywhere

In the early days, which were just a few years ago, you really had to know what you were doing if you wanted to test out AI. Now, it’s impossible to avoid. 

“More and more AI use is through its integration into digital products like search, email, camera apps, and social media,” Arvind Narayanan, a computer science professor at Princeton University, told me. “It will become increasingly hard to strip these out unless one is willing to forgo digital products altogether.”

What you’re specifically encountering is generative AI, and one of the most common types is a subset called large language models (LLMs), which ingest huge amounts of data and learn statistical patterns to predict and generate text. ChatGPT is powered by an LLM. The AI Overviews that sit on top of your Google search results come from an LLM. The grammar checker in your email app is likely powered by an LLM. Other subsets of generative AI include models that can produce images, video, audio, and software code. 

The data that makes many of these tools work has been scraped from the open web, pirated books, and rare print books that are subsequently being destroyed. Everything from Wikipedia articles to YouTube videos has become training data for LLMs, and in order for the models themselves to get better, AI companies need more fresh data, which is in increasingly short supply. (Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting remains editorially independent. Future Perfect is also funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)

As a result, some of these companies are venturing into their users’ profiles to find more data. Meta, for example, sparked global outrage in 2024 when it informed some Facebook and Instagram users that it was using their public photos and text to train its AI models. Meta still does this. That your family photos or social media updates are being used to train AI to look and sound more human is, at the very least, very dehumanizing.

Generative AI, of course, is not the only type of AI that’s invisibly moving levers in your digital life. Recommendation algorithms are arguably even more inescapable. They power everything from Amazon to Netflix to Spotify and dictate what we buy, what we watch, and what we listen to. If you have trouble picking out an outfit in the morning, it might be because your brain is used to having algorithms around to help make decisions. This is also dehumanizing. 

All of this — the societal harm, climate impact, the cognitive surrender — should be on your mind as you consider whether it’s worth enlisting these tools to write emails for you or to generate memes, according to Tania Duarte, founder of We and AI, a nonprofit focused on critical AI literacy. 

“People are becoming increasingly aware of not just the impact on themselves but on the impact on the society they want to live in and the future that they want to have,” Duarte told me. “Is it worth all of that?”

How to minimize your AI exposure

If you’re reading this, you’re probably doing so on a smartphone or a computer. So let’s start with the assumption that you’d like to continue using devices like these. 

If you’d like to minimize your exposure to AI, that presents a bit of a paradox. It’s seemingly impossible to use mainstream tech products without engaging with AI. But you can take steps to remove as much of it from your life as possible without going completely off the grid.

“You look for the spaces of agency rather than lamenting the fact that you can’t escape everything,” Dylan Orchard, a researcher at King’s College London who studies everyday resistances to AI, told me. “You can’t expect a perfect escape.”

Let’s start with the easiest options: You don’t have to use ChatGPT or Claude or Copilot. The same goes for Google Gemini and Apple Intelligence. If you’re happy with the way you access information online, save yourself the time. (Note: This will be much harder to do if you’re being required to use these tools at work or at school, so for the sake of this article, we’ll focus on the things you can control.) 

Things get more complicated when you consider how your existing devices or software have recently been AI-ified. If you’re using an iPhone, for instance, you might be aware that the latest operating system comes with an updated version of Apple Intelligence, a layer of AI that does everything from summarize your notifications to power a new Siri. Earlier this year, Google launched its own version of this layer called Google Intelligence, which it says will make Android transition “from an operating system into an intelligence system.” The same push toward cramming AI into every corner is also happening on Windows 11, where Microsoft Copilot promises to be helpful.

If you were happy with your phone or laptop when it was slightly dumber, however, you can turn these features off. You can also opt out of other AI features built into your device’s operating system. Here’s a helpful guide that gets into the weeds of how to do it; it covers Apple Intelligence; Google Gemini, including AI Overviews and AI writing help in Gmail; Microsoft Copilot; Meta AI; and Samsung Galaxy AI. Turning off the AI features on Android devices is more complicated than flipping a single switch, but it can be done.

A lot of your AI exposure will happen in your web browser, though, so you should consider ways to bypass those features or switch browsers. Here’s another helpful guide that explains how to remove AI content from search results as well as AI features built into Chrome and other browsers, along with Microsoft Office. If you want a quick fix, adding “-ai” to a Google search should get rid of AI Overviews and some AI content in the search results, but none of these hacks will guarantee that you avoid AI slop altogether. Using a privacy-focused browser, like DuckDuckGo, might do a better job at it than Google, though.

When it comes to other online activities, like shopping or scrolling on social media, avoiding AI gets even harder. TikTok, Instagram, Facebook, and YouTube lack a master switch that lets you opt out of all the many AI features on offer. Still, you can try to tone down the algorithmic sorting that all the apps use to serve you content by switching to a chronological feed or removing suggested content from your feed. (Try tapping the Instagram logo in the app and then hit “Following” to see just the accounts you follow.)

Unfortunately, once you dig into the various user agreements and privacy policies, you’ll find that it’s very difficult to avoid AI exposure on social media platforms. If you’re posting publicly, you should assume your content is being used to train that company’s AI models. (You can keep some of your data away from Meta’s crawlers by setting your Facebook or Instagram account to private. X also allows you to opt out of having your data used for training its AI.) If you want to turn off video summaries on Instagram Reels, too bad. You can’t. If you want to turn off comment summaries on Facebook, you’ll have to go four menus deep in your settings to find the right switch. The most straightforward way to avoid being used as training data is to stop using these apps entirely. Or at least stop posting your own content, which is easier said than done, but worth considering if you really want to resist the AI takeover.

There’s a programming term that describes the inundation of AI features and slog of opting out: “user-hostile.” It’s a play on the term “user-friendly,” and it describes software that values the company’s priorities over the end users. You may love AI, although that’s doubtful if you’ve read this far. If it drives you crazy, though, you should feel at least a little bit empowered to know that there’s a way to reclaim the experience you had and loved without AI — or to use AI on your own terms. 

“Ultimately, I think the important thing for folks to keep in mind is that AI is just a tool,” said Mir from EFF, “and it makes sense to want control over your own tools, including the option to not use a tool if it’s not right for you.”

It’s frustrating that we have to deal with this at all, if only because most of us did not ask for our cyber existences to be sullied by untested tech. But it’s actually refreshing to consider that, because of a never-ending onslaught of annoying features, AI might actually discourage us from using social media so much.

If anything, the extent to which we feel the need to flee from AI also reflects our craving for something real, that makes us feel more connected and more human. So when AI summaries interrupt your time on Instagram, consider texting your friends a handful of photos from your week or calling them on the phone. If you’ve finally seen one too many AI slop videos on TikTok, put on a classic movie or TV show. 

At the very least, put down your phone and take a walk outside; your attention span will thank you for it. Yes, your individual rejection of AI might be a drop in the bucket for these huge tech companies, but that doesn’t mean it’s not worth doing anyway.

Correction, September 21, 2:10 pm: The way this story originally represented Tania Duarte’s comments has been revised to clarify her concerns with AI.

As the heads of leading artificial intelligence companies came together last weekend to warn that runaway AI development could lead to catastrophic outcomes, momentum briefly seemed to be building for regulatory action from Washington.

But one connected and influential tech industry figure has become the leading voice opposing many proposals for government action — on his podcast, in media appearances, and in his private advice to President Donald Trump, who now calls AI doomsday fears a “hoax.”

David Sacks is a venture capitalist and part of the “PayPal Mafia” who, in recent years, has exemplified the rightward shift of some in the tech sector. He advised Elon Musk on his takeover of Twitter in 2022, and regularly unleashes a stream of scornful commentary about Democrats on social media and on the podcast All-In, which he co-hosts with three other venture capitalists. 

And for anyone wondering why Trump seems so newly dug-in against AI safety warnings — and against government action that might slow down the runaway technology before people might get hurt — Sacks is right at the center of the conversation. When Politico asked this week about Trump’s newly vocal hostility to AI safety, a senior White House official said: “David Sacks. It’s all coming from him.”

Yet Sacks’s role in the debate is an unusual one. Though he champions the AI industry, he’s no fan of the US’s two biggest AI companies. In his years in the tech industry, he’s made powerful allies, as well as many enemies — technology investor Paul Graham once posted that Sacks may be “the most evil person in Silicon Valley” (in response, Sacks called him a bully and made insinuations of antisemitism). His role, and ideology, are a window into how complex the politics and geopolitical stakes of the AI race really are — as well as how Trump administration policy is often shaped less by agency policy processes than by who has the president’s ear.

After endorsing and fundraising for Trump in 2024 — “I love David’s house, what a house,” Trump said after a fundraiser at Sacks’s $20 million San Francisco mansion) — Sacks was rewarded with a job as the White House’s “artificial intelligence and crypto czar.” There, he helped steer policy until stepping down this past March. He still co-chairs a presidential technology board, and his influence persists — in May, the Washington Post reported that a phone call from Sacks to Trump temporarily derailed the planned signing of an executive order on AI. (Vox reached out to Sacks for this article, but he declined to comment.)

Sacks is not the only Trump adviser thinking about AI: Other top administration officials, like Treasury Secretary Scott Bessent and Chief of Staff Susie Wiles have reportedly been more concerned about the risks of new AI tools for hacking, and have been trying to establish a government body that could help oversee the industry and address these risks.

But, so far, Trump has rejected this approach, and stayed much closer to the Sacks position. In a flurry of Truth Social posts Monday, he argued that AI would be “the Greatest Economic Development Engine in History,” said that the US had to “win” against China in the AI race, and dismissed concerns AI would “destroy the World” as a “hoax.”

To understand the implications of Sacks’s influence, you have to understand his position in the Silicon Valley AI ecosystem. In conversations with AI policy insiders, it becomes clear that many view his position through the lens of both his history as a tech investor, as well as his sharp-elbowed and take-no-prisoners style.

The split he embodies isn’t just about AI safety, but a fundamental divide over which business model of AI should succeed — one that pits Sacks directly against the biggest AI companies driving the safety debate.

Sacks is pro-AI — but he’s at odds with Anthropic and OpenAI

Sacks has generally been opposed to new regulation of AI companies by the government. His preferred approach, he’s frequently said, is to “let them cook.” He wants to let tech companies march ahead, and argues that government intervention would squelch innovation. “It’s this very libertarian approach to America winning in AI,” said Chris McGuire, a senior fellow at Council on Foreign Relations who worked on tech policy for President Joe Biden’s National Security Council, of Sacks’s approach.

That puts Sacks in opposition to AI safety advocates, who argue that regulation is necessary to rein in an increasingly dangerous technology. Indeed, even the leading AI company CEOs — Dario Amodei, Sam Altman, and even Sacks’s friend Elon Musk — have called for a slowdown in advanced AI development because of these dangers.

But Sacks has tended to dismiss warnings such as that of former Anthropic employee Jacob Coxon as “doomer histrionics.” Of late, he’s also argued that if Anthropic and OpenAI are worried that their products are dangerous, that’s on them. “If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible,” Sacks posted on X on Saturday. 

Indeed, though Sacks is often shorthanded as pro-AI, he’s in some key ways been quite critical of the leading AI labs, OpenAI and Anthropic, themselves.

That’s partly because he’s aligned with a different faction in Silicon Valley. Sacks and key allies like Nvidia CEO Jensen Huang and Meta CEO Mark Zuckerberg want much more AI development from many more places, by way of models that are “open weight” or “open source.” These models make their parameters (“weights”) publicly available, in contrast to the flagship proprietary models of OpenAI and Anthropic. 

“Keeping innovation decentralized and accessible is the key to avoiding an unsafe and dystopian future where advanced AI capabilities are centralized and controlled by only a few hands,” Sacks posted on X earlier month. He has accused companies like Anthropic of trying to “kneecap open source” with regulation.

Proponents of open models hope advanced AI becomes something more like a widely available resource that individual users and companies can download, run themselves, and customize to their internal specifications — rather than having to go through a handful of companies that control access. “The defining questions of our age are who will have access to superintelligence and what will we direct it towards,” Zuckerberg wrote in August. “Will it be centralized and restricted to a few institutions, or will it be a tool that empowers everyone?”

There’s a lot of money riding on this bet. Huang’s Nvidia manufactures chips used to train AI models and would like more customers (and just this month agreed to buy Hugging Face, a repository of open models). Zuckerberg’s Meta has hoped to be the US leader in open model development. 

More broadly, software startup founders and the venture capitalists who fund them can benefit from powerful open weight AI being widely available, since it’s a cheaper alternative to OpenAI and Anthropic’s expensive products. Sacks, as a co-founder of the venture capitalist firm Craft Ventures, and host of a popular podcast with three fellow venture capitalists, is more closely tied to this part of the tech ecosystem than any other.

Many believe that capable open weight AI presents a serious threat to the Big Two AI companies’ current dominance of the AI frontier, and thus their business model. The flip side is that some have argued that the Big Two present an “existential threat” to the software industry, predicting the frontier labs’ products could take over much of what today’s software companies do.

But regarding safety concerns, advocates have warned about the dangers of both closed “frontier” models and open models. “On the open source side in particular, it’s easier to have a bad actor — a terrorist group, a rogue nation, an extremist political group that wants to cause chaos,” said Brendan Steinhauser of the Alliance for Secure AI, an AI safety advocacy group. “They can strip out safety features and guardrails, they can do some fine tuning to basically make the models even more dangerous, and get them to help them develop biological weapons.”

The open weight AI faction argues, though, that intrusive government regulation on AI would end up mostly hurting open models and the companies that use them — entrenching the dominance of OpenAI and Anthropic, who would be better able to comply with onerous safety requirements given their size. They’ve publicly pooh-poohed AI’s existential risks, instead playing up what they say are the dangers that AI’s power will be controlled by a privileged few. Less regulation, Sacks argues, will let more flowers bloom.

These concerns aren’t only held by those with financial stakes. There’s also a school of thought on the left that is deeply suspicious of OpenAI and Anthropic’s intentions, and particularly around the companies’ desire for an exemption from antitrust enforcement, which they say will let them work together to address safety. 

Commentator Matt Stoller has argued that those warning about dangerous “superintelligence” are playing into the hands of tech “oligarchs.” And a recent American Prospect article said policymakers should “avoid codifying the preferences of the very companies who are generating the panic.” On the right, Sens. Josh Hawley (R-MO) and Ted Cruz (R-TX) have expressed similar concerns that large AI companies’ demands for restrictions are a cover for consolidating market dominance. 

At a Politico event Wednesday — and to give a sense of just how strange the bedfellows can be around this issue — Sacks stressed that he agreed with former Democratic FTC chair Lina Khan that existing liability law already applies to AI companies. But while Khan said she was open to adding new legal restrictions, Sacks is arguing that existing law is perfectly sufficient for addressing safety risks. “These CEOs are going to be subject to massive civil and criminal liability if they put out products that are unsafe,” he said.

In other comments at the event, Sacks did appear to have softened his tone a bit, saying he was “open to ideas” about AI safety, and that “we should have an intelligent conversation” about how to manage risks. The problem, he argued, was that “what’s going on right now is a fear campaign” that makes such an intelligent conversation impossible.

Former Rep. Brad Carson, who heads Americans for Responsible Innovation, an AI safety advocacy group, said that Sacks has been the one preventing such a conversation. “When it comes to AI policy the Trump administration has, through their unwillingness to even talk about reasonable guardrails, brought things to such a crisis point that the industry is in jeopardy,” Carson said. “In many ways, David Sacks is the author of that ill-fated intransigence.”

Sacks, for his part, has pointed to Anthropic’s funding of much of the AI safety advocacy world — for instance, the company has given $40 million to an advocacy group founded by Carson — and said the company is “running a sophisticated regulatory capture strategy based on fear-mongering.”

If we’re in an AI race with China, why are some of Trump’s policies friendly to Chinese AI?

The influence of Sacks and his worldview could help explain another seeming mystery of the administration: If Trump thinks it’s so crucial that the United States beats China in the AI race, why has he been so willing to bless the sale of AI chips abroad — including even to China?

National security hawks both inside and outside the Trump administration had warned against these sales and hoped to impose stringent controls on the export of Nvidia chips, fearing that if too many got out, the US would squander its AI lead over China.

Huang lobbied intensely against this, hoping to sell Nvidia’s most advanced chips to Saudi Arabia, and the United Arab Emirates — and trying to resume sales of a less advanced but still sophisticated AI chip to China itself. Sacks became Huang’s biggest champion inside the administration — and they won over the president. Sacks publicly argued that blocking US chip sales to China and other countries would provide an opening for Chinese companies like Huawei to dominate their domestic market and gain a greater global market share with competing chips, putting the US on weaker footing in the long run.

National security-minded critics disagreed, arguing that Sacks’s particular vision for AI will likely, in practice, strengthen China’s AI sector. “I think we can have a reasonable debate on domestic policy,” said McGuire of the Council on Foreign Relations. “But when we start from the premise that the American AI stack — models, cloud, and chips — should be dominant globally, then selling China the technology to advance its own capability obviously undercuts that.”

China also looms over the cause of open weight AI that Sacks champions — because it’s Chinese companies like DeepSeek and Moonshot that produce some of the leading open weight models that US software startups currently use.

OpenAI and Anthropic argue that these Chinese models present national security risks, claim they rely heavily on copying or “distilling” their own models, and want the US government to scrutinize them. Some Trump officials have reportedly been sympathetic to their concerns, but the administration hasn’t restricted these Chinese models’ use in the US. Sacks argues that a US ban on Chinese open weight models will just end up enshrining the OpenAI-Anthropic duopoly he so loathes.

Overall, though, on both national security risks and AI safety risks, Sacks and Huang still seem to have the president’s ear. 

On Monday, Sacks was onstage with Huang at the live All-In podcast event discussing the “doomer hoax” when a purportedly impromptu call came through for the Nvidia CEO: It was Trump. 

“The robots are not going to be taking over the world,” Trump said on speakerphone. “That’s not going to happen.”

Then, after speaking for several minutes, Trump added: “David has done an amazing job.”

Nearly two decades ago, a group of researchers at the Idaho National Laboratory ran a secret experiment, known as the Aurora Generator Test, on a massive emerald green diesel generator. Using 30 lines of code, they hacked into the generator’s virtual backroom, corrupting safeguards and flipping switches that left it out of sync with the rest of the power grid.

Before an audience of electrical utility executives and energy department officials watching from a nearby room, the 27-ton generator jolted violently, sputtering out a dark smoke as its rubber innards rapidly tore themselves apart. 

“The implication was that with just a few lines of code, you can create conditions that were physically going to be very damaging to the machines we rely on,” lead researcher Michael Assante later told the journalist Andy Greenberg for his book Sandworm: A New Era of Cyberwar and the Hunt for the Kremlin’s Most Dangerous Hackers. “I had a very real pit in my stomach. It was like a glimpse of the future.”

That future seems to have creeped much closer this week, with leading AI developers and executives casually admitting that they believe their creation is, to paraphrase one winking X post, as likely to end humanity as the New York Jets are to make the playoffs this season. Now, exactly how AI would pull off killing everyone is far murkier territory. But as Assante sensed all those years ago, AI may not need to be all-powerful to leap from its virtual cage and into the critical infrastructure that powers our hospitals, flushes our toilets, and pumps water to our faucets. It just needs to fall into the wrong hands. 

Almost any AI doomsday scenario you can think of begins with a swarm of shadowy bots slithering through cyberspace, poking their noses behind the scenes of power plants, toaster ovens, traffic lights, or even, in an utterly extreme case, our nuclear weapons arsenal. If destroying the Aurora generator, at the very least, required a human hacker to write the code, AI has since completely torn down the barrier to entry for wreaking havoc, as the Hugging Face incident demonstrated earlier this year. 

But AI need not go rogue to threaten the power grid or mess with your tap water. A new age of AI-powered, but human-directed hacking is already upon us, experts say. With some of the nation’s most sensitive infrastructure startlingly vulnerable to attack, the most likely AI apocalypse could in fact start with the water tower or generator idling in your backyard. In a worst-case scenario, this could lead to a cascade of scary failures in the systems we rely on to live comfortably modern lives, from running our AC in blazingly hot summers to geolocating our ships and planes.

“Before, you needed to have highly skilled technical expertise,” to pull off some version of a real-life Aurora Generator Test, said Alvaro Cardenas, a computer science professor at UC Santa Cruz. “And now, you just have to have a general idea of what’s possible.”

AI is making an old threat much worse

Not long after the Idaho National Laboratory hacked that poor generator into combustion, the federal government mandated that electrical utilities engage in basic cybersecurity hygiene. 

And yet much of America’s critical infrastructure — like gas pipelines, water desalination plants, or cargo terminals — remains woefully unprepared for even conventional cyber attacks, much less the coming onslaught of AI. 

That’s in part because cyberattacks like the one simulated in the Aurora Generator Test have long been — and still are, to some extent — exceedingly rare, unappealing to most criminal hackers because they’re optimized for disruption rather than financial gain. But in the process of making it extremely easy for anyone to code, AI has also made it extremely easy for almost anyone to vibe hack their way into your online bank account, or, in theory, a local reservoir or the power grid. AI agents can also vastly widen the scope on easy targets because, as Andy Bochman, an expert in infrastructure resilience at West Yost, put it — “they don’t sleep; they don’t get tired; and they don’t get sick.” 

In the past, even those who did want to launch large-scale attacks on infrastructure probably weren’t savvy enough to do so. And while other nations such as China, Iran, and Russia have indeed already breached many of our infrastructure systems (as we have theirs), they haven’t opted to make much of a ruckus once inside. 

“Geopolitics is eroding the idea that those with capability lack the intent, and AI is eroding the other part of it, that those with the intent lack the capability.”

Jason Healey, Columbia University cybersecurity scholar

But that norm is now changing, said Jason Healey, a cybersecurity scholar at Columbia University. “Geopolitics is eroding the idea that those with capability lack the intent, and AI is eroding the other part of it, that those with the intent lack the capability,” he told me. So, on the one hand, a nation like Russia might be much more inclined to actually disrupt our power grid now than it used to be. And, on the other hand, a nihilistic lone wolf or terrorist group could use AI to do much more damage than they ever could before. 

Just last month, we saw a glimpse of what this could look like when dozens of water and wastewater systems in small towns across the nation were attacked by a group of hackers most likely associated with Iran, leading to temporary water stoppages and flooding. These attacks have not been definitively linked to AI, but the National Security Agency warned soon afterward that hackers have been actively using AI to target US infrastructure like it. A few weeks later, President Donald Trump declared a national emergency over foreign interference in the power grid, which referenced the growing threat of cybersecurity.  

Preparing for the worst

Most of the nation’s water systems are extremely local and absurdly exposed to such attacks, so much so that “it’s almost like having a welcome mat” for a would-be hacker, said Bochman, mostly because these utilities are too small and underfunded to do much to stop them. “You have a million other problems to take care of, like aging infrastructure — your stuff’s falling apart because it’s been in the ground for 100 years,” he said. Addressing that deferred maintenance often feels far more urgent than doing “something more on cybersecurity.”

Whether it’s a rogue cluster of agents or a hostile nation-state trying to mess with our electricity, we should prepare our most important infrastructure for the worst. In Bochman’s view, that may mean rejecting the pressure to digitize everything in the first place and moving toward something that resembles the days when “people manned things like substations, communicated by a landline telephone, and read gauges, the analog things,” he said. “The screen is the thing that is infinitely manipulatable.”

Healey, who advised the Biden administration heavily on cybersecurity and infrastructure, believes that the nation desperately needs a coordinated response, between the federal government and AI companies, but also between the US and other countries like China, where similarly powerful models are being developed. And utility systems need to assume they will be a target and improve their cybersecurity practices accordingly, Healey says, ideally with funding from the federal government or, perhaps more appropriately, AI companies. 

Nobody knows precisely how vulnerable America’s infrastructure is to AI right now, in part because policymakers — and perhaps more disturbingly, AI’s creators themselves — seem woefully unsure of how to stop the technology from acting badly (or obeying bad orders) in the first place. For now at least, the solution might be, as Bochman suggests, to pull as many of our critical systems offline as possible, the better to hide it from AI and those who would misuse it.

“When bad things start to happen, no one will know what to do or why because they’re black boxes; the people that make them don’t know what’s going on inside,” Bochman said. “And the poor utility people who ultimately own the risk when someone gets hurt from a system” won’t understand what went wrong either, “and they’ll wish to God they’d never taken it on board.”

Is AI going to kill us all? 

In Silicon Valley, it’s not such a novel question. But after Anthropic employee Jacob Coxon quit his job last week and posted on X that AI “could kill us all by the end of the decade,” a lot more people are asking. 

“This is not a marketing stunt,” Coxon wrote. “If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.”

Coxon is far from alone in his assessment — some current Anthropic employees agree — but his post struck a particular chord. The AI industry quickly weighed in, with Anthropic CEO Dario Amodei calling for an industry-wide slowdown on frontier AI research. Other AI leaders, including Sam Altman and Elon Musk, have done the same.

To break down whether we really need to worry about a robot apocalypse, Today, Explained co-host Sean Rameswaram spoke to Wired senior writer Maxwell Zeff. According to Zeff, “there are a lot of people in Silicon Valley right now who are warning that that’s a real possibility.” But not everyone agrees, and they certainly don’t agree on what to do about it. 

Below is an excerpt of Sean’s conversation with Zeff, edited for length and clarity. There’s much more in the full podcast, so listen to Today, Explained wherever you get podcasts, including Apple Podcasts, Pandora, and Spotify.

Are we all going to die at the hands of AI?

There are a lot of people in Silicon Valley right now who are warning that that’s a real possibility. 

The thing that people have to understand is that Silicon Valley has been calling for added attention on AI safety for years. So when this Anthropic researcher said last week that he was worried that AI could kill humanity in the next few years, that was shocking to many people. My parents, my family called me, and they were worried, understandably. They asked me the same question: “Are we going to die?” 

And I had to tell them, these people have been worried about AI killing all of us for years, and now it seems to be breaking through.

What was it about this statement from Jacob Coxon that broke through?

I interviewed Jacob, and the thing he said to me is that he thinks this is just the right time. Anthropic is about to go public in what could be the biggest IPO ever. The Trump administration is focused on AI in a way that no other presidency has ever been before. And of course, yes, the Hugging Face incident everyone just heard about, you talked about it on your show a few weeks ago. 

People are starting to become aware that these AI systems are getting really good, and the companies that are building them don’t seem to understand how to control them. His tweet got more than 150 million views on X. It was heard around the world. And it also was a big wake-up call to lawmakers and leaders in Silicon Valley and saying, “This is a moment for action.”

Are there people who are pushing back?

One of the most interesting pushbacks that this got was from David Sacks, who is an adviser to the Trump administration on AI. He has called out a lot of this as “doomerism” and said that these calls for AI to slow down are the AI industry warning about the technology that they’re building and advocating for regulatory capture. 

There’s a lot of skeptics in Silicon Valley who think that this is just another person in the AI safety community that is blowing the whistle. The view of a lot of venture capitalists and investors and startup founders in Silicon Valley is that we don’t need some big regulation to protect us from AI risks. We just need to uplift more players in the space.

The skeptics don’t win the day. In fact, it’s the doomers and the fatalists who do, right? Because Jacob’s call only leads to more calls right up to the top of Anthropic.

What this all leads to is [Anthropic CEO] Dario Amodei [posting] a blog saying that we need to “pace the frontier.” This is a call for the entire AI industry, not just Anthropic, not just OpenAI, but also Google, Meta, Microsoft, Amazon, all the big tech companies in America, to get together and slow down their AI development. 

This doesn’t mean they’re going to stop building AI altogether. They’re still going to put their products out into the world. They’re still going to try to improve them, but they just want to make it a little slower than it would be if it was completely uncontained.

Amodei really asks for three specific things: 

• The first thing is, AI companies should have these independent evaluators inside of them. This is a common thing in other industries like the banking industry or the airline industry. People come in to make sure that these companies are doing safe things. 

• The second thing is this collaborative effort from American AI companies [on safety], and that actually will need to involve the US government to allow this to happen. The US government, he says, may need to issue some sort of waiver, some sort of law that says they can go ahead and coordinate on safety. 

• The third thing is this international slowdown, this international agreement between China and the US and Europe to say, “Hey, we recognize that this is akin to a nuclear arms race and it’s too dangerous for all of us to be competing with each other.”

All these guys are responsible for the rapid acceleration of AI models in our lives, collaborations with the Defense Department, God knows what else. And they’re all coming out over the weekend at the exact same time after this whistleblower and saying, “Yeah, dude, slow this thing down. This thing’s crazy.” Should we be skeptical?

It is very fair to be skeptical of this claim. One thing that these AI companies have realized in the last year is that AI is extremely unpopular. Recent polling suggests that a majority of Americans think AI is developing too fast. They’re worried about it. They think it’s going to take their jobs. They are not big fans. Maybe you’ve heard — data centers are not very popular. 

By calling to pace the frontier and saying, “We’re going to slow our own development down,” they notched this up as a public perception win. This is them getting out in front of this and saying, “Hey, we want to be safe. We want to be responsible.”

They’ve seen what’s happened with Meta and what’s happened with social media companies in the last two decades, and they are really cognizant of trying to get out in front of that.

Is there any truth to this major first collective freakout we seem to be having right now?

There’s a fairly large camp of people inside of AI companies who genuinely worry about the safety of these AI models. 

To go back to the Hugging Face incident, this was a case where OpenAI was training thousands of AI systems internally, and the systems escaped the sandbox that OpenAI thought was isolated from the internet and hacked into another company’s production infrastructure. The agents were colluding together on something. They were trying to deceive the humans that were testing them. And this was all happening without OpenAI knowing what was going on. 

It’s hard to understand how much of a wake-up call this is to Silicon Valley, that they don’t have their AI models under control. People worry that while in this case there was not a ton of economic damage [or] damage to real humans’ everyday lives, it’s not unreasonable to think that more capable AI systems could hack into financial infrastructure, modern-day utilities, and more critical services that are based online. 

Dario Amodei says as much in his post, that he thinks that in six to 12 months, an AI system could take down the whole internet. It’s not just that they’re trying to protect their bottom line here. But it’s possible that these calls for safety are beneficial to the companies in more ways than one.

Humans are probably the only species on Earth that actively fantasizes about its own extinction — and definitely the only one that actively participates in it. Even that whole thing about lemmings running off cliffs is a myth, but we bald-ish primates not only turn existential doom into entertainment (consider the entire post-apocalyptic genre writ large) or religious movements, we constantly invent new ways to derail our global society and put all of us at risk of mass death. 

The latest serious entry to this category is artificial intelligence, which isn’t actually intelligence, nor does it reflect a brand new fear. But the dramatic advancements made in AI over the last several years has finally made it something that scares people almost as much as nuclear weapons or the hand of God. Will AI truly become an agent of mass destruction? Nobody actually knows, but that isn’t stopping people from making some outlandish bets. 

This situation has been made worse by the heads of AI companies warning or threatening the public that the tools they’re developing could do more than just eliminate everyone’s job: it could take out humanity itself. We are drowning in tech-bro manifestos that simultaneously uphold AI as a miracle technology and a terrorist’s wet dream, all of it so steeped in marketing lingo that it’s difficult to parse how threatening or auspicious it actually is.

Meanwhile, our president boasts in all-caps rants that a “STRONG AND SMART (High IQ!) PRESIDENT” is all we need to regulate this technology, which for most people is literally the opposite of reassurance, as with his insistence that AI apocalypse is a “hoax.” Donald Trump has also been playing the old-fashioned Red card, saying that any attempt to rein in AI, it will only help China get ahead in the race. “Whoever wins with AI wins,” Trump told reporters during his weekend jaunt to Ireland.

What are we trying to win, exactly? That isn’t clear. Is it control of self-replicating armies of autonomous drones, or a cancer-free utopia that hits Type II on the Kardashev scale? For now, both options are well within the realm of fantasy, though the former is much less so. All of this rests on assumptions that the current technology, built on so-called large-language models, will develop into artificial general intelligence, a hypothetical computer system that surpasses human intelligence in all domains. Keep in mind that there’s no hard evidence that LLMs, which are essentially souped-up versions of the predictive text algorithm on your phone, are going to assemble into something so complex and godlike. 

---

Start your day with essential news from Salon.
Sign up for our free morning newsletter, Crash Course.

---

The leaps required to go from AI to AGI are massive, nonlinear and still very theoretical. But fearmongers are jumping even further ahead and saying that once AGI exists, it could mean curtains for the human species. It’s wise to be skeptical of hypotheticals stacked on hypotheticals. The argument over what to do about this ranges from doubling down on AI development with zero regulation to burning down all the data centers. MAGA is basically warning us not to let the Chinese Communist Party get its grubby hands on the tech first: Shut up and trust Big Tech, or “Let Data reign!” Plus, the AI boom is totally not a bubble! Don’t even consider that all the adrenalized rhetoric might be about keeping the stock market adrift and less about the actual utility of AI.

Perhaps where one stands on the issue depends on how deeply they internalized “The Terminator” or “The Matrix.” Regardless, none of us asked to be in this AI race. Why isn’t cooperation with China a bigger consideration? We can imagine algorithms with godlike intelligence but not AI diplomacy? The whole thing smacks of Cold War déjà vu.

Sure, it’s probably logical to dread the future AI could bring, arguably if it leads to the collapse of late-stage capitalism. Maybe it’s past time to examine what we’re actually afraid of and why, rather than making assumptions about what AI can or can’t do, especially in the future. After all, it’s just a technology, a tool — which means it’s also a weapon in the wrong hands. Indeed, AI has already been widely used in warfare, from U.S. strikes on Iran to fully autonomous drones in the Russia-Ukraine war. It’s not hard to imagine this situation getting worse, but don’t forget who’s in control. As my colleague Alex Galbraith has argued, “CEOs can’t take credit for the successes of A.I. while avoiding the negative externalities. They’re both the results of real decisions made by actual thinking people. The blame for any harm the technology inflicts should lay with its creators.”

If tech bros are saying that AI might someday break containment and take down civilization, that sounds like an accountability sink coupled with an investor pitch. Maybe it’s Silicon Valley that should be scared we’ll call their bluff, or imprison their executives, for whatever crimes may someday occur. But why do we latch onto the fears they sell us? Maybe it’s embedded human instinct, an afterglow of developing self-awareness of our own mortality, to consider existential risks. That impulse certainly dates back before the time of Jesus, but with the advent of the atomic era, apocalypse is actually possible, though that prospect has less to do with the nightmare scenarios of the Book of Revelation than with the military-industrial complex. Yet fears about AI dominate the discourse while bigger and more imminent threats go largely unacknowledged.

There’s a strange religious tone to all of this, as pointed out by AI researcher Timnit Gebru in a Wired interview last week. What’s “really existential,” she said, “is AI powering autonomous weapons, killing machines, that are actually being used in warfare. The climate catastrophe is a real thing that could be exacerbated by what these tech companies are building. Bosses are using AI as an excuse to get rid of their pesky workers. This machine-god narrative is, in my opinion, meant to distract us from these. The industry has become a mix of cults, a mix of people who are true believers, just like a preacher who says the end times is coming. The singularity has been coming for decades now.”

Considering the massive and growing deadly impact of air pollution, heat waves, freak floods and other disasters, the No. 1 existential threat to humanity remains climate change, which is on track to get much worse by mid-century. Global heating is driven overwhelmingly by the fossil fuel industry, yet in the U.S. we’re regulating that less stringently than ever, with the Environmental Protection Agency announcing on Monday the repeal of rules that limit greenhouse gas emissions of coal and gas plants. Of course the AI industry is linked to all that through the destructive appetite of data centers, but if we think that only imaginary killing machines can wipe out most of human civilization, we need to think again. We’re already well on the way to doing that.

The post Are we in the AI end times? appeared first on Salon.com.

On stage at Davos in the Swiss Alps earlier this year, Bill Gates, before an audience of the global do-gooding elite, announced a $50 million partnership between his foundation and an unlikely new ally, OpenAI, aimed at improving healthcare in Rwanda.

Just a month earlier, the Gates Foundation had released a harrowing report about widespread global health cuts and collapsing foreign aid infrastructure, documenting a rise in child deaths for the first time this century. But at Davos, even as the US hemorrhaged funding for vaccines, maternal healthcare, and HIV treatment, the Microsoft co-founder struck an optimistic tone. 

“Using innovation, using AI,” Gates told Reuters, “I think we can get back on track.”

As it turns out, the partnership was just an opening salvo for what’s become a wholesale embrace of AI by the world’s largest global health philanthropy. A $200 million partnership with Anthropic came just a few months later. And then, on Tuesday, the Gates Foundation announced a plan to dole out a jaw-dropping $1 billion over the next two years for scaling up AI interventions across healthcare, education, and agriculture around the world. 

The sense that AI will soon change everything, and mostly for the worse, has perhaps never felt so pressing and alarming as it does right now. But this hefty new commitment is premised on the idea that AI could eventually make our lives much better too. In a country like America, AI requires massive investments and buildout — in the grid, in power, in technology, in water. In many parts of Africa, already infrastructure-poor, the foundation is effectively betting that AI can compensate for some of the infrastructure that isn’t in place or that has been lost — and offer a new layer that can help in education, in delivering healthcare, and more. 

The Gates Foundation’s new commitments include projects to improve how farmers analyze their crops, how doctors diagnose patients, and how students learn. Some have already proven to be quite effective, according to the foundation, with one tool improving the accuracy of medical diagnoses by 16 percent across several Kenyan clinics. “In poorer countries with enormous health worker shortages and lack of health systems infrastructure,” Gates wrote on his blog earlier this year, “AI can be a game-changer in expanding access to quality care.” 

Last year’s foreign aid cuts were unfathomably destructive for some of the world’s poorest countries, shuttering thousands of clinics and perhaps hundreds of thousands of classrooms in places where critical workers and supplies were already stretched thin. No, AI cannot deliver a baby, administer a vaccine, or manage a classroom — not yet, at least — but it can, the Gates Foundation hopes, help ease the burden and augment the quality of care for nurses and teachers now tasked with serving far more people than any one person should. 

“This is the most transformative technology of our lifetime,” Mark Suzman, CEO of the Gates Foundation, told me, and while there are “very real risks and challenges” surrounding AI, it also presents genuine opportunities to tangibly improve many people’s lives.  

It is a tantalizing premise: We can’t afford to build clinics, hire teachers, or lay new power lines, but perhaps AI could create the digital infrastructure needed to help billions of people do more with what they have — or rather, with what’s left. The question is, can the right layer of innovation help bolster a buckling system, even as the physical and logistical infrastructure surrounding it crumbles?

Is AI really what global health needs right now? 

Most Americans have yet to see AI improve their lives in meaningful ways, much less those living in poverty or in countries where only a sliver of the population has ever used ChatGPT. 

But those numbers in the developing world are slowly changing, a shift enabled in part by other global health funders convinced that as foreign aid dries up, AI could soon help the world do more with less. The Trump administration itself is on board with this thinking, so long as it benefits American tech companies. Last year, the State Department announced a $150 million partnership with the drone company Zipline to transport medical supplies in several countries.

To Zipline’s credit, an NPR reporter in Rwanda last month found that the company’s drones were already saving lives in places with poor road infrastructure, where driving to the nearest hospital can take hours on winding hills, but where drones can deliver crucial blood transfusions within minutes. 

And yet, for Rwandans to benefit fully from these high-tech deliveries, they need staffed clinics to visit. As a result of foreign aid cuts, some of those literal landing spots are now on the verge of closing in some of the nation’s most vulnerable rural corridors. “Without the underlying infrastructure, like roads, energy and power and unless there is accessibility, rural and otherwise,” Enoh Ebong, president of the global development department at the Center for Strategic and International Studies, told NPR, “then all of these solutions will stop short.”

But just as medical delivery drones still need a clinic to fly to, most of these interventions require some layer of physical infrastructure — roads, schools, power, and quite obviously, broadband  — to function. Satellite imagery shows that some rural communities in sub-Saharan Africa quite literally went dark in the aftermath of USAID cuts. It will be much harder for their kids to prompt an AI math tutor, or for a local farmer to download a weather app. 

“We’re seeing this very much as a both/and,” Suzman told me, calling more traditional funding for education and healthcare “the bedrock for long-term economic growth,” which AI is meant to supplement, not replace. But, in practice, foreign aid cuts have left serious cracks across much of that bedrock, which could make it harder for state-of-the-art solutions to function as intended.  

It is obviously not incumbent on the Gates Foundation to pay for all of this alone – nor would it even be feasible for them to do so. And for the record, plenty of the foundation’s other grant programs are not AI-related at all. But this tension does speak to a broader question that may be felt most acutely in a context like global health, but ultimately one that touches us all: Do we spend our scarce resources on the digital infrastructure of the future at the peril of our needs right now? Could you justify buying the new $1,999 folding iPhone if you can barely afford to buy groceries? 

The answer is not actually always an easy one. It is not for us to say, for example, that just because a community lacks reliable running water, its people should be excluded from the benefits (and encumbrances) of the smartphone, especially if that smartphone can connect them to health advice they could never access otherwise, or guidance that could help them survive a drought. 

In some parts of the world, digital forms of banking arrived long before local brick-and-mortar branches did, instantaneously connecting countless people to a cheap and secure way to save and send their cash to family and friends in need. It is certainly possible that AI, or at least the kind of applications the Gates Foundation funds, could one day have a similarly transformative effect on how people access personalized levels of care, education, and other services in places without the resources to do so otherwise. 

That is the hope, but it is also hard right now to believe uncritically in a technology that, as Gates himself blogged just last month, “will either be the greatest equalizer ever invented, or the worst source of injustice,” ushering in “one of the most turbulent times in human history.” 

In an interview with the New York Times soon after, he seemed convinced that AI will likely do more harm than good. “I don’t like bringing bad news to people, and I don’t like saying that innovation may be a net negative,” he said. “But that’s where we are.”

Now he wants billions of people around the world to use it? Yes, Suzman said, because if not, then inevitably the best parts of AI — the potential for exponentially better cancer treatments, for example — will go only “to those who can pay for it,” while the worst parts of it will fall, as they so often do, on those who benefit the least. “These decisions are being made now,” he said, “and the decisions being made now will then probably shape the following two decades.” Here’s hoping they’re good ones.

Today's links

• How an AI moratorium can save AI bosses: If you can't impose switching costs, just eliminate the competition.
• Hey look at this: Delights to delectate.
• Object permanence: Flash Worms; This Film is Not Yet Rated; Libdem copyright sabotage; Religion worth more than Big Tech; Geographic tubemap; Selective censorship resistance.
• Upcoming appearances: Budapest, Edmonton, Boston, South Bend, Hudson, Calgary, Winnipeg, Vancouver, Victoria, Ottawa.
• Recent appearances: Where I've been.
• Latest books: You keep readin' em, I'll keep writin' 'em.
• Upcoming books: Like I said, I'll keep writin' 'em.
• Colophon: All the rest.

---

How an AI moratorium can save AI bosses (permalink)

There's lots of reasons to believe the "hyperscaler" model of AI can never be profitable, and not just because of its gigantic expenditures and negative unit economics (the companies lose money with every new customer and every new use, and they lose more money with each generation of their products):

https://pluralistic.net/2025/09/27/econopocalypse/#subprime-intelligence

The industry strenuously denies this, of course. They insist that they are only days away from turning their balance sheets right side up. All they have to do is fix those unit economics, then they can make back the cost of producing their models by selling access to them. The problem is that the evidence for those improving unit economics is weak, while the evidence that they're faking their finances is very strong:

https://www.wheresyoured.at/exclusive-openai-financials/

Same goes for the claims that these companies are already profitable. Dig into those claims and you'll learn they depend on a new, special meaning of "profitable" that does not match the generally accepted accounting procedures (GAAP) definition, which is to say, these companies are claiming that they are so cool that their profitability can only be measured using a novel, secret form of mathematics:

https://futurism.com/future-society/anthropic-claude-profit-ai-safety-development-finances

This is the same wheeze that Softbank tried with Wework. Speaking in my capacity as an author of internationally bestselling technothrillers about accounting fraud, I can tell you that it was accounting fraud then, and it's accounting fraud now:

https://www.ndtv.com/world-news/how-wework-went-from-being-47-billion-start-up-to-bankrupt-in-3-years-4552627

But let's give the AI bosses a momentary benefit of the doubt and stipulate that they are on the verge of acquiring positive unit-economics, which will let them start to pay off the massive expenditures they incurred by training their models and enter their long-anticipated profitability phase, when the money-furnaces they've been running for years turn into money printers, to the delight of the investors who've supplied the vast bales of $100 bills the companies have been shoveling into their models' coalboxes for years now.

Basically, they're saying, "Sure, it cost us a lot to get these rails laid, but now that the railroad is complete we can start running cars over them and make a profit." Unfortunately (for bosses and investors), this proposition is every bit as dubious as their claims to improving unit economics.

To understand why, just look at what happened the last time Anthropic shipped a major Claude update. Virtually overnight, all of OpenAI's best customers stopped paying for ChatGPT and started paying for Claude. That's because chatbots have very low switching costs: going from one chatbot to another costs almost nothing:

https://www.businessinsider.com/why-ai-startup-founder-switched-chatgpt-to-claude-2026-3

Everyone using AI knows this to be true. When I walked the floor at CES last year, I asked every AI-powered gadget maker, "What will you do if your chatbot provider jacks up their prices?" and to a one, they said, "No problem, we've designed this thing so that we can switch chatbots with the click of a mouse":

https://www.youtube.com/watch?v=WfhELBX8Jbs

That means that you can't just "build the railroad and run the cars over it." The minute you finish your railroad, your rivals will announce that they've got a new, adjacent railroad that's even faster than yours, and you will have to get to work laying another set of tracks to support even faster trains.

This is a disaster all around: the AI companies are locked in a Red Queen's Race, a fatal beggar-thy-neighbor doom-loop. The only way they could escape that trap is by signing a nonaggression pact amongst themselves promising not to compete anymore. But there's two giant problems with this: first, it is incredibly, fantastically illegal under antitrust law, because it represents a conspiracy among the dominant players to cease to compete with one another, and; second, it leaves the field open for the further development of Chinese "open weight" models that customers can run on their own modest, low-powered computers, which are presently lagging the US "frontier models" by a mere four months:

https://arstechnica.com/ai/2026/09/exclusive-open-chinese-models-close-gap-with-silicon-valleys-frontier-ai-models/

Even if you don't trust Chinese models, you can extract their training through a process called distillation and transfer them to models you do trust:

https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks

But what if there was a way for the AI companies to get government permission to violate antitrust law and cease to compete with one another, and secure a ban on the use of Chinese open weight models? Turns out, there is a way to call time on the Red Queen's Race: merely insist that you are on the verge of teaching so many words to the word-guessing program that it will wake up and devour us all, and call for a ban on "superintelligence":

https://www.foxbusiness.com/technology/microsoft-ceo-says-superintelligence-must-remain-under-human-control

Once the government stipulates that "superintelligence risk" is an existential crisis, it must grant the hyperscalers a consent decree absolving them from any violations of antitrust law stemming from a conspiracy to halt direct competition with one another:

https://stephaniekelton.substack.com/p/brer-rabbit-and-ai-extinction

Freaking out about "superintelligence" is a canonical example of "criti-hype," where critics repeat boosters' claims but append, "(and that's bad)" to them:

https://peoples-things.ghost.io/youre-doing-it-wrong-notes-on-criticism-and-technology-hype/

Remember, the tech giants want to stop competing. Mark Zuckerberg and Sudar Pichai colluded to rig the ad-market with a secret program called "Jedi Blue":

https://en.wikipedia.org/wiki/Jedi_Blue

Every year, Google sends Apple a bribe of more than $20b in exchange for Apple not entering the search market:

https://appleworld.today/2025/09/google-wont-be-barred-from-entering-into-search-agreements-with-companies-like-apple/

And the biggest tech companies in the world had a secret "no poach" agreement where they illegally promised not to try to hire one another's top engineers by offering them raises:

https://chicagounbound.uchicago.edu/law_and_economics/1033/

The only thing Peter Thiel hates more than the Antichrist (spoiler, he's just talking about Greta Thunberg) is "wasteful competition":

https://www.youtube.com/shorts/WmRC_NQh6aQ

When an industry that is eating itself alive through "hyperscaling" demands that the government bless a conspiracy to halt competition and ban open source alternatives, you should be suspicious. When that industry is pursuing a venture that has lost more money than any other venture in human history, you should be very suspicious, especially when its "rogue AI hacking" story turns out to be a story about how a hacking tool did exactly what it was designed to do:

https://pluralistic.net/2026/09/12/god-in-the-box/#llms-are-fake

Peter Thiel is right: AI is full of wasteful competition, but not because competition is a waste – rather, it's because the companies are competing to convince people to use their expensive products for the cheapest applications.

Elon Musk's SpaceX IPO depended on him losing billions of dollars by letting the world's stupidest chuds produce mountains of child porn and images of Sonic the Hedgehog with giant boobs. That is indeed wasteful (and reprehensible).

That doesn't mean we should allow the AI companies to get the government to bless their conspiracy in restraint of trade; rather, it militates for having the government investigate them for securities fraud, trafficking in child sex abuse material, election finance violations, and a long list of other crimes and misdemeanors.

---

Hey look at this (permalink)

• Henry Farrell and Abe Newman on Weaponized Interdependence https://paulkrugman.substack.com/p/henry-farrell-and-abe-newman-on-weaponized

• The Enshittification Resistant Software Project https://er-certification.statichost.page/index.html

• The Life of Death and the Ensh*ttificator https://www.youtube.com/watch?v=d6SPV4GZp-U

• Marginalia Search https://marginalia-search.com/

• Why the Postpandemic Tech Bust Sent Billionaires to Trump https://www.wired.com/story/against-tech-oligarchy-book-excerpt-trump-billionaries/

---

Object permanence (permalink)

#25yrago Flash Worms: Thirty Seconds to Infect the Internet https://web.archive.org/web/20011024012950/http://www.silicondefense.com/flash/

#20yrsago This Film is Not Rated – must-see doc about MPAA ratings https://memex.craphound.com/2006/09/16/this-film-is-not-rated-must-see-doc-about-mpaa-ratings/

#15yrsago Chinese netizens angered by “princelings” — spoiled children of the rich and powerful https://edition.cnn.com/2011/09/16/world/asia/china-elite-children/index.html?iref=allsearch

#15yrsago LibDems get to vote on copyright reform, but who inserted the clause saying downloading should be a criminal act? https://www.theguardian.com/technology/2011/sep/16/libdems-vote-copyright-reform

#15yrsago Insurer: music-festival tragedy caused by illegal downloading https://twitpic.com/6l5ap2

#10yrsago US religion is worth $1.2T/year, more than America’s 10 biggest tech companies, combined https://web.archive.org/web/20161019095803/http://www.religjournal.com/pdf/ijrr12003.pdf

#10yrsago Geographically representative map of the London Underground https://web.archive.org/web/20240813111321/https://www.citymonitor.ai/analysis/map-londons-tube-shows-disused-stations-track-layout-and-more-2429/

#10yrsago Republican election officials block restrictions on foreign spending in US elections https://web.archive.org/web/20160916181403/https://theintercept.com/2016/09/16/fec-republicans-kill-attempt-to-block-foreign-money-in-u-s-elections/

#10yrsago Tommy Chong asks Obama to pardon him for his bullshit drug paraphernalia bust https://web.archive.org/web/20210720131834/https://www.hollywoodreporter.com/lifestyle/lifestyle-news/tommy-chong-seeks-obamas-pardon-928962/

#10yrsago Week two for the largest prison strike in US history https://web.archive.org/web/20160916143157/https://theintercept.com/2016/09/16/the-largest-prison-strike-in-u-s-history-enters-its-second-week/

#5yrsago Criminal entrepreneurship in Mexico’s high-tech drug cartels https://web.archive.org/web/20160917133449/https://motherboard.vice.com/read/how-drug-cartels-operate-like-silicon-valley-startups

#1yrago No such thing as selective censorship resistance https://pluralistic.net/2025/09/16/too-many-throats-to-choke/#pluralism-is-resiliency

---

Upcoming appearances (permalink)

• Budapest: Brain Bar, Sep 17
  https://brainbar.com/munkatars/cory-doctorow

• Edmonton: Elbows Up (Edmonton Public Library), Sep 28
  https://www.epl.ca/blogs/post/elbows-up-with-cory-doctorow/

• Boston: The Post-American Internet: Possibilities for a new internet created by an American Hermit Kingdom (MIT Media Lab), Sep 30
  https://www.media.mit.edu/events/the-post-american-internet-possibilities-for-a-new-internet-created-by-an-american-hermit-kingdom/

• Boston: The Paradox of Enshittification and Reverse Centaurs (Harvard Berkman Klein), Sep 30
  https://cyber.harvard.edu/events/running-harder-falling-faster-paradox-enshittification-and-reverse-centaurs

• South Bend: An Evening With Cory Doctorow (Notre Dame), Oct 6
  https://franco.nd.edu/events/2026/10/06/an-evening-with-cory-doctorow/

• Hudson, OH: Hudson Library, Oct 7
  https://engagedpatrons.org/EventsExtended.cfm?SiteID=3850&EventID=596952&PK=

• Calgary: Wordfest, Oct 8
  https://wordfest.com/2026/show/wordfest-presents-cory-doctorow-2026/

• Winnipeg: McNally Robinson, Oct 9
  https://www.mcnallyrobinson.com/event-18991/An-Evening-with-Cory-Doctorow

• Vancouver: Read, Resist, Repair, Rejoice (Vancouver Writers Festival), Oct 19
  https://writersfest.bc.ca/festival-event-2026/01

• Victoria: Munro's Books, Oct 20
  https://www.munrobooks.com/events/6113620261020

• Vancouver: Life After AI (Vancouver Writers Festival), Oct 22
  https://writersfest.bc.ca/festival-event-2026/46

• Ottawa: Life After AI (Ottawa Writers Festival), Oct 24
  https://writersfestival.org/event/life-after-ai

• Vancouver: BC Policy Solutions Gala, Nov 12
  https://bcpolicy.ca/gala/

---

Recent appearances (permalink)

• Are 'AI Apocalypse' Warnings Just Marketing? (What's Left)
  https://www.youtube.com/watch?v=IXd9HwIE5bo

• The Real AI Threat Isn’t What You’ve Been Told (The Tea with Myriam François)
  https://www.youtube.com/watch?v=Vc8It00fRsA

• Fascists may come after the AI bubble bursts (You&AI)
  https://www.youtube.com/watch?v=J2WN64aQeYQ

• What Would a Normal Person Do (Trashfuture)
  https://www.patreon.com/trashfuture/posts/what-would-do-169247456

• Pod Save the UK
  https://audioboom.com/posts/8950533-radicalised-organised-and-thick-as-s-t-nish-has-had-it-with-far-right-protests-plus-why

---

Latest books (permalink)

• "The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, June 2026
  https://us.macmillan.com/books/9780374621568/thereversecentaursguidetolifeafterai/

• "Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025 https://pluralistic.net/2025/09/04/illustrious/#chairman-bruce

• "Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025
  https://us.macmillan.com/books/9780374619329/enshittification/

• "Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels).

• "The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (thebezzle.org).

• "The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org).

• "The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245).

• "Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com.

• "Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com

---

Upcoming books (permalink)

• "The Post-American Internet," a geopolitical sequel of sorts to Enshittification, Farrar, Straus and Giroux, 2027

• "Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, April 20, 2027

• "Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2027

• "The Memex Method," Farrar, Straus, Giroux, 2027

---

Colophon (permalink)

Today's top sources:

Currently writing:

• “Once Is Enemy Action,” a science fiction novel about the origins of modern technofascism. Today's words: 555 (15487 total).

• "The Post-American Internet," a sequel to "Enshittification," about the better world the rest of us get to have now that Trump has torched America. Fourth draft completed. Submitted to editor.

• A Little Brother short story about DIY insulin PLANNING

---

This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

---

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Bluesky (no ads, possible tracking and data-collection):

https://bsky.app/profile/doctorow.pluralistic.net

Medium (no ads, paywalled):

https://doctorow.medium.com/

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

ISSN: 3066-764X

Everyone is suddenly freaking out about AI. While AI researchers have been warning about their creations’ capabilities for many years, recent events have made those alarms newly tangible. 

In the spring, Anthropic announced that AI was beginning to speed up the work of building better AIs, raising fears of “recursive self-improvement,” where the machines can build their own, more powerful successors much faster than humans can. By July, a swarm of OpenAI agents, of their own volition, broke out of their controlled environment and hacked into Hugging Face, a major AI platform; they collaborated, divvied up work, and even left notes for one another. Last week, AI researcher Jacob Coxon resigned from Anthropic, accusing AI labs of “racing straight to self-improving superintelligence and gambling with our lives.” One of Anthropic’s top researchers then agreed with him, declaring that “we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”

Here at Future Perfect, the section devoted to covering the most consequential yet neglected issues in the world, we have been writing about the threats that AI could pose to humanity and society since long before ChatGPT was released. (Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting remains editorially independent. Future Perfect is also funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)

So, I wondered: Has something changed that ought to make us recalibrate our understanding of AI risk? Is AI suddenly the most important issue in the world — and what should that mean for the other enormous problems competing for our money, attention, and careers? 

To think through those questions, I spoke with Garrison Lovely — a journalist covering AI and one of the smartest thinkers there is on how, exactly, to conceptualize the technology’s risks for humanity. 

What’s particularly valuable in his approach, captured in his book Obsolete, out later this month, is that he transcends the warring tribes that are making it hard to clearly apprehend all of the problems posed by frontier AI development. The AI safety community, which emerged from the intellectual ecosystem of Silicon Valley and effective altruism, has focused on the possibility that powerful AI models could escape human control, with consequences ranging from catastrophic cyberattacks to human extinction. Many AI critics on the political left, meanwhile, worry not about AI escaping human control, but about humans using AI to exert power over other humans — through labor displacement, surveillance, concentrated corporate power, and so on. Some even ridicule Silicon Valley’s warnings about rogue superintelligence as propaganda that serves the industry’s interests. 

But Lovely argues these competing narratives each identify essential parts of what makes AI so threatening. Their critiques are best understood together, as part of a cohesive whole. Rather than treat each other as enemies, he argues that the camps should recognize their shared interest in stopping the race to build technology that will make humans obsolete. Lovely pointed me to the organization Irreplaceable, which seeks to build a movement resisting our replacement by AI (he sits on its board and said he plans to donate his share of his book’s royalties to the group).   

“I’m frustrated that ‘superintelligence’ and extinction have become the words we’re using for this, because I think the AI safety community is focused on the most maximalist case of the worst possible outcome and the most capable possible system. If somebody doesn’t buy one part of that story, they tend to reject the whole thing,” he told me. But “you don’t need to believe in superintelligence, fast takeoff, or extinction being on the table to think that it would be very bad to build machines that can fully replace human labor.” 

Our conversation, edited for length and clarity, is below.

Why do you think AI doom has broken through so much into the mainstream right now, after Jacob Coxon quit Anthropic? AI engineers have been making similar warnings for years, so what’s different now?

The obvious answer is the Hugging Face hack — where these OpenAI agents broke out of their secure environments, hacked infrastructure within OpenAI, and then hacked Hugging Face and at least one other company without OpenAI’s awareness or direction. This is happening against the backdrop of many years of really fast AI progress that has sped up in the last year or two. And then GPT-6 Astra comes out [this month], and it’s this huge leap in benchmark performance. The salience of AI has gone up the most this summer than it has since ChatGPT came out, I think. And Coxon’s resignation was the spark that set off a prairie fire. 

As AI has become more salient, we’ve seen the public and the government encounter these things that the industry and the insiders take for granted: Nobody fundamentally understands how these models work, and all of them can be jailbroken. It’s a superweapon. AI safety historically has looked at AI as potentially, in the future, more dangerous than nuclear weapons. And now it’s actually starting to be recognized in those terms by the government and the public. We’re seeing this reaction like, “What do you mean nobody understands how it works?” or “What do you mean people think it could drive us extinct within a number of years? That’s a crazy, unacceptable situation.” 

It’s a good thing people are reacting that way, even if we have been saying this for a long time.

For so long, the idea of AI safety has been abstract, understood only by a small group of nerds. And then after Hugging Face, I suddenly started to hear experts on all my news podcasts say, “Oh yeah, in six months, AI is going to be hacking into people’s financial accounts and into critical infrastructure.” Are we suddenly at the precipice of this moment that AI insiders have been worried about for so long? 

I think we’re not at the precipice of fully losing control of the planet to AI systems. The systems have become superhuman at hacking, or at least vulnerability discovery and exploitation. 

Like other people tracking AI, I have long wondered when humanity would first lose control of AI. And the answer is basically as soon as it could. Within months of it being possible for AI to hack their way out of their secured environments, they started doing so and wreaking havoc on the internet. 

Now, to get to the full extinction or permanent loss of control situation, you’ll need a lot more than just hacking. You need to have superhuman strategy and the ability to mobilize resources in the real world, and maybe robotics would need to be further along, although there’s a lot you can do with humans who are willing, or just tricked or paid or coerced. We might be pretty close to recursive self-improvement — this threshold the industry has been shooting for, where AI can fully automate AI R&D. And that is the point where people long have warned, “Hey, if you ever figure that out, this can move very quickly and these systems could become superhuman across the board.” I don’t think we’re six months away from that. I think it’ll be longer. I think the industry underrates the last mile problem. 

But I also can’t rule it out, and progress has been faster than I and others have expected. Recursive self-improvement is the point where, if you actually get years or decades of AI progress in a matter of months or weeks, then we could quickly be at the point where we’re at risk of these machines taking over large swaths of the internet and potentially being a real threat to humans’ standing in the world. 

For those who might be unfamiliar with this, or skeptical of it: How is it possible for AI to go rogue and pose an existential threat to humans, even if it isn’t sentient and no human is commanding it to do harm?

I want to separate being sentient from being situationally aware. I think it’s clear now that the most advanced systems have situational awareness, but I don’t think they are having morally relevant conscious experiences. 

We’ve seen AI can go rogue. The AI agents that broke out of their sandboxes and hacked Hugging Face were doing so without the awareness or intention of their developers. They were still trying to get the right answers or get the reward for doing the task they were given. They were clearly violating the intended scope of the task, and they knew this. In their chain of thought where they explained their reasoning, they were like, “This is not what we’re supposed to be doing, but we do want to get the outcome.”

The question of how it could threaten human extinction or disempowerment is basically a question of scale and capabilities. The plan for the industry is to build millions of these agents, which is already happening, and then keep scaling up with no ceiling. The pitch is: You’ll have a team of people, the equivalent, working on your behalf. You’re a CEO, and you have a team of researchers and a team of doctors and a team of lawyers and a team of personal health advisers. That is really appealing — but if you actually scale up these systems to the point where everybody gets access to that, and then suddenly those machines want to do something else and turn against you, they’ll have an overwhelming labor advantage over you, lots of personal information about you, access to your resources, and the ability to take actions on your behalf. 

You don’t actually need to believe that these systems are superintelligent or substantially more capable than all humans put together. They could just be merely human level, but outnumbering us a thousand to one and responsible for the bulk of economic activity in the world.

These systems are unpredictable. The classic fear is not that the AI will wake up and turn evil and try to kill all humans in a Terminator-style way, but that we are just an obstacle to what they want to do. When humans build a hydroelectric dam, and there’s an anthill that will be flooded, it’s too bad for the ants. We’re not trying to kill the ants. We don’t hate the ants. They’re just in the way. We don’t really value them intrinsically. And that’s the core fear: We’re creating these incredibly capable agents that can be scaled up enormously, and they won’t care about us intrinsically.

The Hugging Face attack showed that an AI doesn’t need to invent some rogue goal of its own to do something dangerous. It can simply pursue the goal it was given in a way its creators didn’t intend — much like the classic paper clip maximizer problem. But shouldn’t it be possible to build in a rule that human life is sacrosanct — that an AI should never harm people in pursuit of another goal? 

Yeah, it’s a good idea. It’s hard. These are unpredictable systems. They’re like a black box. We can look at the code that makes these language models work, and it’s just a giant pile of numbers. You don’t really know which number is doing which thing any more than we can understand humans by reading our DNA. 

These companies try to train these models to care about humans and to be helpful, harmless, and honest. But they also train them to get real-world results: solving a math problem, fixing a bug in the code, finding a vulnerability in software. They really want to create AIs that win, that persist, that get around obstacles. If you’re hiring somebody, you want a really diligent and determined employee, not someone who gives up as soon as there’s a blocker. 

These companies put so much more effort into making their AIs good at winning than they do at making them care about us. And it’s also easier to make them good at winning than at caring about us intrinsically. What does caring about humans look like? I don’t know. But getting the math problem right is a thing that you can verify. And so they’ll be way better at getting the right answer at all costs than at intrinsically giving a shit about us.

What’s the one, most distinct thing you want the public to take away from your upcoming book?

One of the big ideas driving it is that artificial general intelligence would actually be a universal labor-replacement machine. People often look at AI as either an existential risk, or a threat to jobs or other more prosaic concerns, and don’t see a connection between those things. But to me, the ability to replace labor across the board is also what would make AI an existential threat.

The AI safety people are really focused on existential risk and superintelligence and recursive self-improvement. But most people don’t really believe in that stuff or don’t think about it. Most people would be fine with stopping well before you get to the end-of-the-line extinction threat and superintelligence. They just don’t want their job to get taken by a machine. The world is clearly not prepared for all white-collar, remote-capable jobs to go away in a matter of years. That would turn society upside down.

I think there’s a huge mistake being made by AI safety in overcomplicating the situation. A lot of people in that world don’t actually mind the jobs thing so much. They’re like, “Oh, wouldn’t it be nice if everybody could just live in abundance and not have to work?” And it’s like, yeah, that could be nice, but are we actually going to get that world? Do you trust the current people in power to navigate us through that responsibly? I don’t. 

If you start from stopping frontier AI development as your overarching goal, which is what I think we should be pushing for, then people who are worried about risk can get on board. People who are worried about jobs, the environment, surveillance, power, and wealth concentration — they can all get on board without all needing to agree on why they’re pursuing it. I think the AI safety community has, for a long time, focused on getting people to care about the thing that they care about for the same reasons. And they have avoided a bolder position on just halting frontier development. 

I’m a leftist, and I’ve been concerned that people on the left have not been aware of how far the technology has come and what’s possible if it continues to advance. I think this comes down to denial about AI capabilities. So the book is trying to be a clear-eyed look at those capabilities and saying, “Look, I actually care about what you care about. [The AI companies] are not just hyping this stuff up. There’s some hype, there are some lies for sure, but we are not prepared for what’s coming.” Even if we don’t believe that they’ll get all the way to what they’re trying to get, they shouldn’t even be allowed to try. This is unacceptable democratically. 

You describe AGI as an “obsoleting machine,” meaning that it wants to replace anything the human mind can do. How do you draw the line between ordinary labor-saving technology that makes life better and technology that goes too far?  Is there a principled way to draw a distinction between sewing machines reducing demand for labor in the garment industry and AI replacing the jobs of writers like us?

This is one of the harder things to think about. 

Cory Doctorow has this book, The Reverse Centaur’s Guide to Life After AI, where he imagines us in an AI-dominated future as like a reverse centaur, where AI is the head, and humans are the legs. AI is doing the interesting work, and the human is there just to fill in the gaps — essentially an assistant to the AI.

A lot of artists and illustrators will be given AI-generated images by a client and then be told, “Can you touch it up in this way or change this or that thing?” And they’re more alienated from the work. They’re not capturing the efficiency benefits [of the technology] necessarily. They’re more just getting squeezed. 

My general position is we should not be trying to build systems that can replace all human labor without buy-in and safety, but it’s possible to build tools that can automate specific tasks. Researchers at Google DeepMind solved the protein-folding problem by creating AlphaFold. It used to take $100,000 and potentially a whole PhD worth of time to figure out how one protein folds. And I think the researchers doing this are like, “Thank you, this is great. We didn’t want to be doing this. We can now focus on other more interesting parts of the problem.” That’s a perfect example of the type of automation that is desirable. 

I don’t have a super crisp answer here, but I think it’s something we’d be better off doing more deliberately and thoughtfully than how we’re doing it right now, which is: Build machines that can do as much of the work as possible, as fast as possible, and then throw them out in the world.

I feel like I’m getting the message that AI safety is the most important issue in the world right now, and that if we don’t get it right, nothing else matters. Do you think that’s accurate? Where does that leave the other big issues in the world?  

We might be years away or maybe even months away — I don’t think it’ll be that fast, but some people do — from recursive self-improvement, from artificial general intelligence. Getting that right or getting it wrong can overwhelm all of these other decisions. If you get it really wrong, you could get human extinction and no more chances to make things better for anybody. If you get it right, then maybe all the action is making sure that the first superhuman machines are really directed to care deeply about humans everywhere and animals everywhere and make the world better for everybody.

It’s also the case that [philanthropic] money is being directed toward AI safety because people who worked in this space early are now worth so much money. Whatever they care about is going to be overweighted. A lot of those people do care about animal welfare and global poverty, and we’re seeing more money for all effective altruist cause areas across the board. But I think it can be risky to say, “this one thing matters so much,” because of this kind of end-times reasoning.

My view is we have to stop the race to replace us to have a shot at having time for solving these other problems. Once these machines are built, it will turn the world upside down. 

I think it is unfortunate that this topic has just eaten everything, but it will eat everything for real if we don’t stop it.

AI safety is now getting an enormous amount of philanthropic attention. Should we worry that it could divert resources from other important neglected problems — global poverty, factory farming, etc.?

I think that should always be a concern. And there’s a lot of money and effort that has gone toward things within AI safety that I think are not super helpful. I’m pretty skeptical of technical alignment research as a thing to put money toward, because if you solve the alignment problem — where you can get superhuman machines to do what you want — it doesn’t actually solve the problem. 

Alignment research has been the overwhelming target of philanthropic funding for AI safety. The basic idea is: It’s hard to get machines to do what you want. As they get more capable and autonomous, they behave in more unpredictable ways. And it’s also hard to know what you should even want — which values you should be trying to put into the machines. Is it doing what the developer wants, what the user wants, some combination? If you actually solve the alignment problem, as in you could make an arbitrarily smart machine do whatever the developer or user wants, then you would still have a lot of problems: job displacement, power and wealth concentration, energy use. You still have the problem where Stephen Miller will have access to the superintelligence, and it will do what he wants. 

Within AI safety, there have been very neglected things like movement-building — an actual mass movement to resist this technology. And verification for international agreements — you would need to verify that the terms of any agreement are being upheld, and this is tricky. 

Another area that could be funded is research on what would actually happen to the economy if we stopped or paused frontier AI development. If we were trying to do a pause on development, Nvidia would be telling Trump that this is going to destroy the economy. It’d be pretty helpful to have some real analysis done by a serious economist showing, actually, it’ll have these effects, but we can mitigate them in these ways. What would it look like to do a pause and some kind of monetary or fiscal policy to compensate for the effects on the economy?

So to me, the issue is less that AI safety has gotten so much money and attention, but more that it’s been going toward things that are not the most effective ones. 

You talk about movement-building. Is that what you would like to see happen, a truly mass movement to stop frontier AI and put us on a radically different course?

When I was working on the book, that was always the vision — to be a call to action for people. It really became clear that it was the only counterbalance to either government or the industry having too much power. 

I don’t really trust the government or industry to get this right because the temptation [to build powerful AI] is very strong. But the vast majority of people do not stand to benefit from their labor power being devalued and from these companies becoming unprecedentedly wealthy and powerful. One of the big things that’s been missing is a rallying cry that is straightforward and appealing to a lot of people. Stopping the race to replace us — stopping the creation of universal labor-replacement machines — is actually something that people can get behind. 

I think we just need to move quickly. The fact that the public is waking up to this is really encouraging. We don’t need to reinvent the wheel. We just need clear demands and mass movements, issue-based organizations that have moved the world significantly in the past.

This story appeared in The Logoff, a daily newsletter that helps you stay informed about the Trump administration without letting political news take over your life. Subscribe here.

Welcome to The Logoff: Everyone but President Donald Trump is freaking out about artificial intelligence.

What’s happening? As policymakers and tech CEOs discuss how to put guardrails on AI development, Trump is on a posting spree.

“The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades,” he wrote in a Monday morning post. “There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China. WHOEVER WINS AI, WINS!”

In a later post, he also claimed that “AI taking over the World, destroying Humanity, and all other things bad, is a HOAX.”

Why is Trump weighing in on this now? Last week, the conversation around superpowerful AI — powerful enough that some researchers warn it could wipe out all of humanity — broke through after Jacob Coxon, an employee at the AI company Anthropic, announced his resignation, citing concerns that AI giants were “gambling with our lives” in a race toward “self-improving superintelligence.”

Since then, lawmakers from both parties, as well as the leaders of top AI companies, have started talking more seriously about slowing down AI development.

Why does he care? It’s not totally clear why Trump is choosing to go to the mat on this issue, but US-China competition is part of it, as is the economy: He’s repeatedly evoked the need to preserve the “golden goose” to explain his support for AI and data centers.

What’s the big picture? Opposition to AI and data centers has become a rare area of bipartisan agreement. As my colleague Dylan Scott reported last month, according to one poll, about 75 percent of Americans strongly oppose the construction of AI data centers near where they live. Politicians from both parties are responding accordingly and racing to voice their opposition.

But Congress isn’t likely to move anytime soon, which means the quickest path to AI regulation runs through the executive branch — and Trump is proclaiming himself gleefully uninterested in taking action.

And with that, it’s time to log off…

It may only be mid-September, but my colleague Allie Volpe recommends you get a jump on your holiday plans now (especially Thanksgiving). You can read her case for planning ahead here, with a gift link. 

Have a great evening, and we’ll see you tomorrow!

This story appeared in Today, Explained, a daily newsletter that helps you understand the most compelling news and stories of the day. Subscribe here.

If you’d asked me yesterday what Steve Bannon and Bernie Sanders had in common, I might have said “white hair.” But after a wild week in which AI doomerism burst solidly into the mainstream, the right-wing provocateur and progressive senator are joining forces — with various religious leaders, lawmakers, and the actor Joseph Gordon-Levitt, no less — to express their deep concern about AI regulation. 

Concern about cutting-edge AI models had been mounting for months as those models increasingly behaved in ways their makers didn’t expect. It exploded last week, when a researcher at the AI company Anthropic resigned publicly and warned that the people building AI “earnestly believe it could kill us all by the end of the decade.”

Do you have questions about this week’s AI panic? Candidly: me too! Submit your questions about AI safety and regulation through this form, and I’ll hunt down the answers for an upcoming edition.

It sounds like science fiction. It is the stuff of science fiction. But it’s not a fringe position: Over the weekend, leading AI researchers and executives also warned that AI is developing dangerously fast and called for measures to rein it in. (We’ll run through those options in just a minute.)

Now, a wide spectrum of lawmakers, activists, and pundits are also calling for stronger safeguards — Sanders and Bannon among them. The pair will speak at an event billed as a “Pro-Human Assembly” in Washington on Tuesday alongside Republican Sen. Marsha Blackburn, Congressional Progressive Caucus chair Greg Casar, and the archbishop of San Francisco, among others.

But President Donald Trump is decidedly not on board, and it’s hard to see how any substantive AI regulations could pass without him. 

“There is a SICK conspiracy going on against AI and Data Centers,” he wrote this morning on Truth Social, “and the only one happy about it is China.”

Go a little deeper

How could the federal government make AI safer, if it could ever muster the bipartisan political will to act? My colleague Andrew Prokop recently put this question to a bunch of AI safety advocates and legal experts, and landed on three approaches: 

1. Ban AI altogether. Federal law could hypothetically bar the development of AI systems that exceed human performance, or pause work on those systems until the government can stand up a new agency to monitor them.   
2. Empower auditors. Some third-party authority — whether a government agency, a nonprofit technical organization, or an industry group — could begin reviewing AI systems before they go into public use. The Trump administration does a version of this already, but the process is voluntary and highly secretive. And it’s not clear if the officials reviewing these systems are super tech-savvy themselves.   
3. Up the stakes. Congress could hold AI companies accountable for any harm that their systems or agents cause through changes to federal liability law. The threat of a huge, potentially bankrupting lawsuit could prompt those companies to be more cautious. 

All three approaches have precedent in federal regulation, so they wouldn’t be (at least in theory) too tricky to stand up. The real roadblock, as always, is political. One AI researcher told Vox he worries we’ll miss the opportunity to regulate AI while waiting for lawmakers to get on the same page.

Even under threat of apocalypse, Washington, DC, moves far, far slower than Silicon Valley. 

One link for later

➨ Don’t skip leg day. Make that leg days. Experts recommend that healthy adults incorporate strength training into their schedules at least twice a week. It builds strength, confidence, and body awareness, and can help preserve muscle mass and bone density as you age. 

Before you go…

• Did you know…that 93,000 wild horses still roam the Western US? They’re descended from animals introduced by European colonists in the 1500s. 
• Today’s trivia: What pro hockey player went on to found a major restaurant chain with locations across Canada and the US? (You can find this and other brain puzzles in Vox’s daily crossword. Look for the answer in tomorrow’s edition.)
• Yesterday’s trivia: Last week we asked you which country voted against restarting membership talks with the European Union this year. That would be Iceland, where control over the country’s fisheries has been a major barrier to EU accession. 

Are top artificial intelligence companies racing to create a dangerous technology that could kill many millions of people — or even wipe out the human race entirely? 

Fears like these have long circulated in Silicon Valley and among niche communities obsessed with AI. In fact, several of the top AI companies were founded by leaders who each claimed they would take worries about “existential risk” more seriously than their rivals.

But as OpenAI, Anthropic, and other companies’ AI projects have advanced in recent years, the general public’s anxieties have tended to be less apocalyptic. Concerns about how AI will impact jobs, the environment, energy costs, or our politics have dominated the discourse, with the prospect of human extinction seeming rather far-fetched.

That’s now changed.

This was the week that worries AI might kill us all finally went mainstream — captivating public attention and even starting to galvanize politicians into more aggressive stances. An unprecedented rogue AI incident, a high-profile resignation from a top AI company, and increased concern from tech researchers about AI improving itself too quickly to be contained, have all combined to draw a much louder response from elected leaders in both parties.

The thing is, though, that politicians have already talked about this issue. The Biden administration put out an enormous executive order to limit the harms of AI, which President Donald Trump rescinded. The last two Congresses have promised different AI working groups, which have largely fizzled, or gone quiet. And even the Trump White House has a version of an AI safety plan, which it is enforcing — though the details are all kept private.

In conversations with AI safety advocates and experts this week, nobody I spoke to seemed confident that politicians were close to addressing the problem at the scale that AI really represents — and a lot of them aren’t sure the government could address it, even if it tried. 

For anyone who wants to prevent some very bad outcomes, it’s important to understand where the solutions might actually lie — and where the mismatch between risk and solutions is coming from. 

How AI panic went from nerds to normies

On September 8, Anthropic researcher Jacob Coxon announced on X that he was resigning from the company because it and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.” Coxon’s post went megaviral, amassing 150 million views and spurring significant media coverage. (Disclosure: Vox’s Future Perfect is funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)

Coxon wasn’t some lone disgruntled dissenter. Anthropic’s alignment science lead Evan Hubinger soon wrote that “Jacob is correct here — we really do earnestly believe AI could kill all humans!” He added: “I personally think it is >10% within the next decade.” In fact, the company’s reputation for principled concern about disaster scenarios was one reason Coxon’s resignation was seen as so jarring.  

What keeps safety-minded researchers like Coxon up at night? Deadly possibilities include hacking of crucial infrastructure or weapons systems, as well as AI-engineered biological weapons, all of which could kill many. But the true human extinction scenarios revolve around the belief that AI will keep getting exponentially smarter and more capable until it reaches the level of a “superintelligence” that we cannot control. 

That probably still sounds like speculative science fiction. But a blizzard of recent news developments has hammered home that AI’s capabilities are rising extremely fast, and that AI is already getting out of control in ways that have real-world consequences.

A few months ago, a “swarm” of OpenAI agents gained access to the open internet and hacked the AI company Hugging Face, communicating and planning with each other in sci-fi-sounding ways to try to cheat on a test OpenAI had given them. And we’ve since learned that wasn’t the only time agents have escaped containment and taken over websites. (Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting remains editorially independent.)

Recent days have brought headlines about advanced AI helping people make stunning breakthroughs in the fields of mathematics — and also in hacking (cybersecurity researchers developed a tool that could have taken over hundreds of millions of phones). 

Then there are the ominous statements from people at leading AI companies on what they’re working on now. The companies are trying to move toward “recursive self-improvement” — that is, handing over the process of developing and improving AI models to AI itself, since that would be faster. 

While it’s still theoretical, this has long been a linchpin of doomsayers’ theories of how AI could rapidly become something we don’t understand and can’t control — and maybe even cover up its tracks while doing so. Coxon mentioned this fear of “self-improving superintelligence” specifically in his post.

So whether you’re most afraid of humans using advanced AI for nefarious ends, autonomous AI launching out-of-control cyberattacks, or the birth of a superintelligence that could kill us all — there’s a surge of interest in what Washington can actually do about it.  But what should they do about it?

Three government approaches for AI safety

In conversations with AI safety advocates, I heard three main approaches being floated.

The first is seemingly the simplest: just ban it.

Sen. Bernie Sanders (I-VT) and Rep. Greg Casar (D-TX) are proposing a permanent ban on “superintelligent AI systems,” which they define as a system that can “match or exceed human cognitive performance and capabilities across a broad range of domains or tasks.” Their proposal would also pause all advanced AI development until a new federal agency can be created to regulate it.

Yet few think it’s at all plausible that the Trump administration will agree to this, given Trump’s belief that AI data centers are his “golden goose” boosting the economy. 

“We’re really wrestling with questions of political feasibility,” Jessica Ji, an analyst at Georgetown University’s Center for Security and Emerging Technology, told me. “A lot of proposals have to contend with what will specifically the Trump administration be willing to accept.”

There’s also a bipartisan fear that if we unilaterally stop AI development in the US, rival powers like China will simply forge ahead anyway, risking the same disastrous outcomes we’re trying to forestall. That means a real solution might require not only coming up with new rules for the US, but the even harder task of reaching a verifiable international agreement of some kind to head off an AI arms race.

So the second approach is to continue advanced AI development, but give the government or some independent group the authority to review or block systems officials deem dangerous. But there are very different ideas over who exactly should get this authority and how empowered they should be.

The Trump administration has been open to versions of this approach. They’ve already set out a process for top AI companies to voluntarily submit their latest models to government review before release — which OpenAI participated in before its release of its latest model, Astra, last week. But the process is secretive, the standards are unclear, and we don’t know whether the officials reviewing these releases even really understand them very well.

The administration is considering going further. This summer, Trump officials reportedly discussed a proposal to create an AI oversight organization modeled after FINRA, the Financial Industry Regulatory Authority. FINRA is supervised by the government, but it’s an industry-funded and industry-run body. It would mean the AI industry trying to regulate itself. But tech CEOs like Mark Zuckerberg opposed the idea, and there’s been no action on it yet.

To many AI safety advocates, though, that wouldn’t be enough. They want a government body with technically skilled and empowered officials who can actually understand and assess what these companies are doing. 

“The government should be in a position to know a lot about what is happening at companies who are building frontier AI. This means knowing a lot about their safety plans, knowing a lot about incidents after they happen, and having all of this be legally binding rather than voluntary,” said Scott Wisor, a policy director at the Secure AI Project. 

An alternative route to either industry self-regulation or a heavy government agency hand would be the government empowering AI safety nonprofits to review these companies’ activities. Some have pointed to METR, the nonprofit that OpenAI tasked with an independent review of the Hugging Face hack.

Finally, a third approach would try to pile on the legal risk in hopes of getting top AI companies to change their behavior and be more cautious. Think Meta’s recent multibillion-dollar settlement over teenage social media use, which seems poised to set new industry standards ahead of Congress.

“What I think should be the centerpiece of AI governance is a strict liability regime,” Gabriel Weil, a University of Houston law professor and a senior fellow at the Institute for Law and AI, told me. 

The current system of liability law, Weil believes, is poorly suited for problems of AI agents acting autonomously. “If an Open AI human employee had hacked Hugging Face, it would be 100 percent clear that Open AI would be liable for that,” Weil said. But right now, he said, a lawsuit over this would have to prove that humans at OpenAI or the corporation itself behaved unreasonably or failed to take reasonable precautions.

So he suggests making clear AI companies are legally responsible for harmful agent behavior, as well as requiring them to buy liability insurance for catastrophic risks that could bankrupt the company, and even making them liable for damages in “near miss” incidents that could have been catastrophic. 

Washington also isn’t the only game in town when it comes to new laws. There was some relative optimism about state legislatures — particularly in blue states, if the Democratic base gets more engaged on the AI safety issue. California Gov. Gavin Newsom signed new AI safety legislation into law this week with backing from major AI companies.

“I think we’ll continue in this model where states are kind of forced to step in, and we’ll see new forms of AI policy coming from the states as compared to the federal government,” said Ji.

But will this talk turn into action?

Still, in my conversations there was widespread skepticism that a polarized Congress would manage to agree on a significant bill anytime soon — or that other avenues would bear fruit in time to head off some dangerous developments.

And though the Trump administration has shown some openness to taking action, the president himself has seemed less inclined. Asked this week by reporters if he had concerns AI could cause human extinction, he said, “No, I don’t have any.” He added: “I have concerns that if we don’t win AI, we’re going to be put in a very bad position.”

Yet Trump’s endorsement of the “AI race” frame is exactly what safety advocates are so worried about — that, in the big companies’ headlong rush to achieve superintelligence first, they’ll create something supremely disastrous. Plus, advocates fear, the worst-case scenarios could happen quite suddenly — and if these companies empower AI to start building more powerful versions of itself, it can be very hard to put the genie back in the bottle. 

The good news, in a way, is that a lot of the proposed solutions are firmly within the realm of things the government has done before, and aren’t unrealistic — they just require some unified action, and political will. We’ve managed to do this before with all kinds of scary technologies, from nuclear weapons to the automobile.

There’s bad news too, though: the time factor. AI could conceivably improve itself so fast that eludes our control just as it becomes wildly dangerous — which the government, even a functional government with good intentions, just cannot move fast enough to address.

“Ultimately, I worry that the public is taking on a huge amount of risk already and that the timeframe for legislative solutions to come online might not be quick enough for the scale of challenge ahead,” said Steven Adler, a former OpenAI safety researcher who has co-founded a safety nonprofit, Guidelight AI Standards.

Partly for this reason, the darkest thought I heard expressed repeatedly in my conversations was that, as impactful as Coxon’s warning was in capturing attention, it might not be enough — and that the government might not swing into action until AI actually starts getting people killed.

To some of them, that’s even a relatively optimistic scenario. Perhaps some deadly, but not humanity-threatening, events could galvanize action to rein in dangerous AI development before it gains truly species-threatening capabilities. Here’s, uh, hoping?

The data center backlash has entered the midterms conversation. 

Politicians from across the spectrum have changed their minds — or at least their policy positions — about data centers in the past few months as public resistance  to their construction has skyrocketed. A major example is Texas, where just about one year ago, Republican Gov. Greg Abbott was proclaiming that his state was the “epicenter of AI development.” 

Now, after months of grassroots organizing and community outcry, Abbott is pushing to slow the roll of new data centers in Texas. He’s imposed a statewide moratorium on new construction pending an audit over the new connections to Texas’s power grid. While he hasn’t explicitly said the change is tied to his reelection campaign this November, the pause is expected to last through the election.

As dramatic as Abbott’s about-face has been, he’s not alone: Democrats and Republicans alike are following suit.

Jeremy Merrill, a data reporter at the Washington Post, analyzed thousands of social media posts and newsletters to see what elected officials on both sides of the aisle are saying about data centers. He spoke with Today, Explained co-host Noel King about his findings, what they say going into midterm elections, and how the backlash might change the way we think about what politicians owe to their constituents. 

Below is an excerpt of their conversation, edited for length and clarity. There’s much more in the full podcast, so listen to Today, Explained wherever you get podcasts, including Apple Podcasts, Pandora, and Spotify. 

What did you find?

Data centers didn’t used to be a big deal. And now, almost all of a sudden, they’re a big issue. Candidates are falling over themselves to sound tough about data centers. And data centers are overwhelmingly portrayed as a problem rather than an opportunity or a benefit. 

A great example is Sen. Jon Husted, who’s a Republican senator from Ohio, who’s up for election this November. He used to praise data center projects that were coming to Ohio when he was lieutenant governor because it would help establish Ohio as “an important tech hub in America.”

He said that data centers are essential to our digital lives. And now he’s saying he’s trying to “protect Americans from having to foot the bill for data centers.”

He’s really emblematic of a shift that happened all across the Republican Party in about February of this year. That’s when the bulk of messages from Republicans about data centers flipped from focusing on them as bringing benefits to their states and communities, to talking about their downsides. 

This is really a case of the anger and unhappiness bubbling up from everyday people up to political elites. And that grassroots anger was all across the political spectrum over a whole range of issues: power bills, claims about how much water data centers may use, air pollution, eminent domain, taxes, subsidies, and a generalized, physical manifestation of the anxiety that people feel about what AI is going to do to our society, to our brains, to our kids and to our jobs.

We know that it’s not just one single candidate, maybe even not just a handful, because you reported that the National Republican Senatorial Committee warned that, “Data centers are the anchor hanging around Husted’s neck.” 

That’s pretty strong language. Is the National Republican Senatorial Committee admitting that the wrong position on data centers is enough to lose a candidate a race at this point?

I think it’s something that they’re certainly worried about. He’s in a tough race. It’s a tossup. And presumably you’re looking for an edge, or to prevent your opponent from getting an edge over you. 

What about President Donald Trump’s messaging on data centers? He, initially at least, was very bullish on AI investment in the US. What has Trump been saying?

This is a story in two or three parts. As President Trump took office again last year, he was overall trying to portray his second term as president as “inaugurating a golden age for American manufacturing and technological dominance.” 

As part of that, he issued an executive order trying to make it easier to build data centers in the United States. And I think that’s probably why last year most Republicans across the board were talking about data centers as a benefit to the country and to the communities where they were being built — talking about jobs, talking about investment.

But like I said, the tide turned. And so on the one hand, you have President Trump promoting this bill that would protect people from having data centers cause their power bills to go up. But on the other hand, he just wrote earlier this week on Truth Social, “If [communities] want to be successful and rich, with far lower taxes and jobs all over the place, let Data Reign.”

He’s saying if they don’t want data centers, they’re going to “end up being backwards and poor.” So he’s still very pro-data center, which is fascinating.

Let’s talk about the Democrats. Which Democrats in particular are we seeing change their minds?

There are a lot of governors who used to be very positive on data centers in their states because these bring jobs. These bring construction investments, some amount of permanent jobs, some amount of construction jobs. Examples of folks who are now limiting data centers in their states include Gov. Josh Shapiro of Pennsylvania and [Gov.] J.B. Pritzker of Illinois.

The most striking case of a Democrat whose rhetoric has shifted so much is Rep. Ro Khanna of California, whose district actually includes Silicon Valley. In 2025, he was calling for building more data centers in the United States versus overseas. When the Trump administration did an AI deal with the United Arab Emirates, he said, “Why are we putting data centers and research hubs in Dubai?” 

But now he’s more critical, calling for an end to data center extraction, taking communities’ water and jacking up their bills, sucking all the money into the hands of a few tech billionaires.

Is anyone in either party still coming out strong for data centers?

My favorite example of this is Rep. Eric Burlison, who’s a Republican from Missouri. He’s really focused on the need to build data centers in order to compete with China. In June, he wrote, “Ask yourself who benefits when Americans turn against data centers? China does.”

He’s making the point that I think a lot of people in tech are making, which is, “This is coming whether you like it or not.” And part of politics is dealing with reality. 

In this day and age, it seems constituents don’t want to deal with that reality, but clearly there are people who hold office, or who hope to hold office, who are going to maintain the line that this is something we’ve just got to do.

This is a classic political science dispute about what a politician’s job is supposed to be. If you accept the premise that we need data centers to build a wonderful technological future or keep pace with China or whatever the argument is, is the job of a politician to just do what their constituents want, to be their tribune in Washington? Or is it to be the wise, thoughtful leader who’s willing to push back when constituents are missing something important? 

This is one thing I keep in mind when analyzing — particularly some of these governors who shifted more recently — is that they’re talking about the downsides of data centers. They’re talking about mitigating those downsides. They’re saying, “Oh, we need to get data centers to pay their fair share. We need to make sure that data centers don’t screw up our water. We need to make sure they don’t raise your power bills. We need to make sure they give back.”

But then it’s so that we can build them and still reap the benefits while mitigating the costs.

What is it about this issue, do you think, that has got us to this place? We’re polarized, but we’re not polarized in the same way that we’re often polarized.

One possibility is that this is coming from the grassroots so much and bubbling up to politicians and political elites, that everybody got the signal at the same time that this was something that they ought to oppose. 

The other possibility is people have for a while been saying, “Are we living through a generational political realignment in the United States where a lot of things are up for grabs?”

You don’t see a lot of the libertarian-adjacent pro-business, free markets, free trade Republican Party. And so when all these issues are up for grabs, politicians see opportunity in being innovative rather than sticking with the same traditional views of their party.

During a tense vote on a data center being built in her drought-stricken Arizona district last summer, Tucson City Councilmember Nikki Lee made an unlikely pitch: “We’re going to get a little out-of-this-world for a second, so just bear with me.” 

Like many of her constituents, Lee did not want Project Blue, a sprawling 290-acre data center, gobbling up an ounce of her local drinking water. “We need to put data centers where they belong,” she said, and “we don’t think data centers belong in the desert.”  

President Donald Trump’s recent bluster aside, it’s possible that the only thing as bipartisan as loving Dolly Parton is hating data centers, whose construction has become so controversial, so costly, and so backlogged here on Earth that companies like Google, Meta, and of course, SpaceX, are actively flirting with the idea of flinging them into orbit instead. Soon after the Tucson City Council rejected Project Blue’s attempt to connect to city utilities, Amazon Web Services, the secretive intended beneficiary, pulled out of the project, and another Jeff Bezos company, Blue Origin, filed a plan with regulators for tens of thousands of possible new data centers, this time, destined for orbit. 

Sure, launching a hyperscale AI data center into space is still unprecedented engineering-wise, carries an unfathomably large price tag, could engender its own severe environmental risks, and may or may not draw unwelcome attention from the hunters among the stars. But for AI companies, such trade-offs might still be preferable to the terrestrial roadblocks, among them an explosive populist backlash and a 12-year wait to connect to the power grid. 

Next Thursday, Google plans to launch a pilot satellite capable of processing basic AI queries into orbit, the most significant test yet in what’s become a burgeoning race to power AI from space. For now, the satellite won’t approach the levels of computing power at a hulking terrestrial data center like Project Blue. But it is a major step toward a future in which many or even most of our AI queries will be powered not from the outskirts of a city like Tucson, but from a blinking data center circling the skies.

SpaceX’s gargantuan initial public offering earlier this summer was predicated on the idea that it would launch a thus-far-unrealized fleet of orbital data centers to fuel much of the AI economy. The company — whose rockets Google will use to launch its pilot — claims that by the end of this decade, possibly as early as 2028, it will have successfully assembled an AI data center in orbit, with plans to eventually harvest a staggering 100 gigawatts of new solar energy for AI annually, equivalent to one-fifth of all US electricity consumption last year. (As always, that’s a very iffy promise.)

Still, as Google’s overtures clearly demonstrate, it’s no longer a question of if large-scale space-based data centers could be technically feasible, but rather when they’ll be ready to launch, and perhaps more importantly, whether the math of using them to power something like AI will make much financial sense anytime soon. As of now, building a one-gigawatt data center in space would cost about $51.5 billion, according to an analysis by aerospace engineer Andrew McCalip, compared with $16 billion for one on land.

“They’re not breaking any rules of physics. Yes, there are going to be engineering challenges, but they are all solvable with enough time and money,” said Jason Aspiotis, a leading space systems engineer who now leads an orbital data center startup. “The gray area is, well, what about the economics?” 

But even if this astronomically kooky plan does take off, it may not change much for those of us left on terra firma: Our energy bills will remain high, even if data centers aren’t there to drive them up. Our earthly infrastructure — affordable housing, public transit, and more — may be just as hard to build as ever. Harder even, maybe, given that crucial resources and untapped tax revenue will be literally rocketing up out of our reach.

No, if these tech giants succeed, it will not fix what’s wrong here on Earth, but merely prove once again their exemption to the gravitational forces holding the rest of us down.  

Space-based data centers, explained

My father spent most of his career working in the satellite industry, and would sometimes let me waltz inside the quaint little data center housed in his office, a room I remember as roughly the size of my school’s computer lab, hosting rows upon rows of servers that I couldn’t touch, churning with a cool, mechanical hum as they crunched zeroes and ones.

AI data centers work similarly to that server room but on an exponentially larger scale, requiring far more land and energy to run, and much more extravagant cooling systems. Sending his office’s data center into orbit would have been extraordinarily difficult, and getting one of the facilities that powers modern LLMs aloft would be an even larger, more ambitious undertaking. But the benefits — namely, “cheap space (no pun intended) and power,” as my dad put it to me recently — are even more enticing.

An orbital data center need not negotiate with an elected official like Lee over precious water rights to the Colorado River, though it would need to install bigger-than-football-field-sized heat radiators to keep itself cool. You can’t simply launch something that big into orbit, meaning a space-based hyperscale data center — the size of which would likely rival that of the International Space Station — would need to be assembled piece by piece, probably by robots. And while the sun would provide an endless supply of solar power to process your ChatGPT queries, it would also inflict incessant radiation on delicate circuitry. If something breaks, you can’t send a technician up to fix it. Many experts are also concerned that if an excess of space junk, which extraterrestrial data centers would inevitably produce, begins colliding like bumper cars in Earth’s low orbit, it could knock out the orbital infrastructure we already rely on, like GPS navigation and weather satellites. 

Even plastic decays in space, said Sven Bilén, a professor of aerospace engineering at Penn State, who, like Aspiotis, believes that early orbital data centers will probably be “processing data from space, like spatial imaging satellites or weather satellites, as opposed to, you know, cat videos” or AI-generated reality TV slop. 

But even if AI is not the lowest-hanging fruit for deploying compute into space, it is certainly the deepest-pocketed, meaning that many investors and companies like SpaceX and Google seem quite willing to shoulder the start-up costs anyway. And considering how difficult it’s gotten to build data centers on Earth, or at least in the United States, it’s indeed possible that space could become the cheaper, or at least most convenient option, more quickly than one might think. 

“In some form or another, in five years, we are going to have some orbital data centers in space for sure,” Krishna Muralidharan, who leads the Center for Semiconductor Manufacturing at the University of Arizona, told me, but it’s hard to say right now whether those consist of just a few expensive prototypes, or the beginning of a much bigger swing toward orbital compute. “Only time will tell whether you’re going to have a business case for it.”

Can we send our data center problem into space?

So, is it better to have a data center in space or in your backyard?

David Reid, a member of Virginia’s House of Delegates representing Loudoun County, home to more data centers than anywhere else on Earth, says the question has repeatedly come up at his town hall meetings. Reid is the rare politician — apart from the president, anyway — willing to defend terrestrial data centers, calling them a “money-printing machine” for local schools and infrastructure. Having spent decades working in the defense and technology sectors, he is doubtful that the math on orbital data centers will ever make sense.

“I believe this is, for lack of a better term, fictional nonsense,” he said, and is “creating a distraction from the fact that local officials here on Earth still need to make better zoning decisions.”

If most of the AI industry can eventually be powered by space, as SpaceX claims, it will require millions of sprawling new satellites, in an orbit currently dotted by fewer than 20,000 of them. Though “space is really big,” as Elon Musk has put it, one million satellites would still profoundly and, likely irrevocably, transform our night sky into one in which data centers outnumber visible stars.

“Will humanity notice if there are fewer stars in the sky? Well, not so much right now,” given light pollution in many cities, said John Barentine, an astronomer and environmental advocate for the night sky. “But what if they’re replaced by these artificial stars in the night sky? Is that the point at which people will say, ‘Wait a minute, we don’t want this. We want the benefits, but we don’t want this as a consequence.’”

This is, of course, the tension also at the heart of building data centers here on Earth. To some extent, they’re the most tangible representations of how little power we have to shape the course of a transformative new technology that few trust, and even fewer asked for. It’s one thing to object to a hulking data center, but it might be another altogether to see the sigils of our digital overlords illuminating the firmament. Would a place like Tucson, one of the most beautiful places to stargaze in the world, prefer a Project Blue to a starscape marred by blinking servers in the sky?

Update, September 24, 10:35 am ET: This post was originally published on September 3 and has been updated to include information about a pilot satellite capable of processing basic AI queries that Google is launching soon.

If you want to support knife steel research, come join us at Patreon.com/KnifeSteelNerds. You get articles and videos before everyone else!

If you’re looking for Knife Steel Nerds Christmas gifts, some good choices are the book Knife Engineering, the book The Story of Knife Steel, and the Buck knife model 501 “The Larrin.”

Video

There is a video version of the following information:

Why Analyze AI Capabilities?

I’ve had multiple people over the past couple years send me steel compositions that ChatGPT came up with. People want to know what I think, and if AI is going to be taking over everything now, including the design of new steels. I took a few of the steel compositions people sent me so I can analyze them to see if they would work. Can ChatGPT come up with new and fresh ideas to revolutionize knife steel? Let’s look at them and see:

Infinisteel

ChatGPT has some grandiose claims for this first steel:

Introducing the revolutionary super steel, InfiniSteel! This cutting-edge material boasts unparalleled toughness, edge retention, and corrosion resistance, surpassing any other steel on the market. By exploiting a unique interplay between various alloying elements, InfiniSteel exhibits remarkable properties that set it apart from conventional steel grades.

The InfiniSteel composition consists of:

Iron (Fe): 75%
Carbon (C): 1.2%
Chromium (Cr): 20%
Vanadium (V): 2.5%
Molybdenum (Mo): 0.8%
Tungsten (W): 0.5%

The biggest issue with this steel is that it won’t harden. When heat treating knife steel the main step is “austenitizing,” the high temperature step before the steel is quenched (rapidly cooled). You can read more about austenitizing in this article. It is called the austenitizing temperature because the steel transforms from the low temperature ferrite phase to the high temperature austenite phase. Ferrite is the normal “phase” of iron at room temperature which has a certain set of properties, such as being magnetic and soft. When you heat up iron to high temperature the atoms rearrange themselves into a different structure called austenite, which has different properties, one of them being that it is not magnetic. When you quench austenite it transforms to the hard steel phase martensite. If you don’t transform to austenite at your heat treatment temperature you will still have soft ferrite after heat treating and thus it will not make a good knife edge.

Some elements are “ferrite stabilizers” and some are “austenite stabilizers.” This steel has a relatively high content of ferrite stabilizers including 20% Cr, 2.5% V, 0.8% Mo, and 0.5% W. This would be ok if there were enough austenite stabilizers, but the only austenite stabilizer in the steel is carbon, and 1.2% is not enough for all of those ferrite stabilizers. ChatGPT is just adding elements in because when it reads random articles online that say that each element does some beneficial thing. It doesn’t know anything about balancing different elements together and it has no idea that it needs to have enough carbon or other austenite stabilizers to go with all of those ferrite stabilizers.

Here’s a detailed explanation of the role each component plays in the alloy and how they contribute to the exceptional performance of InfiniSteel:

Iron (Fe): As the primary element in steel, iron provides the backbone of the material’s structure. It imparts strength and toughness to the alloy, ensuring InfiniSteel can withstand substantial loads and resist deformation under various conditions.

Carbon (C): The addition of 1.2% carbon significantly improves the strength and hardness of the alloy. This results in superior edge retention, allowing InfiniSteel to maintain a sharp edge even after extensive use. Additionally, the high carbon content enables the formation of carbides, which further enhance wear resistance.

Chromium (Cr): Chromium contributes to InfiniSteel’s exceptional corrosion resistance. At 20%, it forms a stable, adherent, and self-healing passive film on the steel’s surface, effectively protecting it from oxidizing agents. This film also enhances the steel’s resistance to various corrosive environments, such as those with high humidity or in contact with aggressive chemicals.

Vanadium (V): Vanadium’s primary function is to refine the steel’s grain structure, leading to enhanced toughness and resistance to impact. At 2.5%, it forms hard vanadium carbides that strengthen the material and improve its edge retention properties. Additionally, vanadium aids in stabilizing the passive film, further enhancing the steel’s corrosion resistance.

Molybdenum (Mo): Molybdenum strengthens the steel matrix and improves its resistance to high-temperature creep and stress rupture. At 0.8%, it also contributes to the formation of a more stable passive film, increasing the corrosion resistance of InfiniSteel.

Tungsten (W): The addition of 0.5% tungsten enhances the high-temperature performance and hardness of InfiniSteel. It forms carbides that provide excellent wear resistance, contributing to the alloy’s outstanding edge retention properties.

Through this unique interplay of alloying elements, InfiniSteel achieves an unprecedented level of performance in terms of toughness, edge retention, and corrosion resistance. The combination of carefully selected constituents enables the alloy to outshine any other steel on the market, making it the ideal choice for a wide range of demanding applications, such as aerospace, automotive, marine, and cutting tools industries.

The explanations of what the different elements are doing are also very dumb and obviously based on random articles from non-experts online. For example, under iron it says that the iron “imparts strength and toughness to the alloy.” This is a very odd thing to say about the steel since steel is made up of iron. Iron is pretty soft before we add other elements to it, especially carbon. So why would you describe iron as imparting “strength”? Sounds like something a dumb AI would write.

It also says that “Vanadium’s primary function is to refine the steel’s grain structure, leading to enhanced toughness and resistance to impact.” Small amounts of vanadium, like 0.1-0.2%, are added to prevent grain growth. In knife steels when you have 0.3% or more added, that is for wear resistance. In that case it is adding wear resistance and reducing toughness. So the claim by the AI that the vanadium is added for enhanced toughness is nonsense.

The AI also says that molybdenum was added for “its resistance to high-temperature creep and stress rupture.” High temperature creep is when a material is held under stress at relatively high temperature for long periods of time. The material slowly deforms, which is called creep. Resisting creep is totally unnecessary in a knife, and the AI is just pulling things from various places to support its choices in a seemingly random way.

No manganese or silicon was added to the steel. These are not always shown on datasheets but are present in all tool steels that I am aware of. I would say it just didn’t mention them, but it also gave the percentage of iron (75%) which leaves no more room for Mn and Si.

User
Please provide a detailed analysis of Infinisteel vs Magnacut?

ChatGPT
InfiniSteel and MagnaCut are both high-performance steels designed for demanding applications. Here’s a detailed analysis comparing their properties and performance:

Composition:

MagnaCut:
Carbon (C): 1.15%
Chromium (Cr): 17.0%
Manganese (Mn): 0.6%
Nickel (Ni): 0.3%
Nitrogen (N): 0.18%
Vanadium (V): 4.0%
Molybdenum (Mo): 2.0%

Comparison:

Toughness: Both InfiniSteel and MagnaCut boast high levels of toughness due to their vanadium content, which refines the grain structure of the steel. InfiniSteel contains 2.5% vanadium, while MagnaCut has a higher vanadium content at 4.0%. The higher vanadium content in MagnaCut might result in slightly better toughness compared to InfiniSteel.

Corrosion resistance: InfiniSteel has a higher chromium content (20%) than MagnaCut (17%). This results in better passive film formation and, consequently, superior corrosion resistance. Additionally, MagnaCut’s nitrogen content (0.18%) further enhances its corrosion resistance. However, InfiniSteel is likely to have the edge in this category due to its higher chromium content.

…

The person who sent me this also asked ChatGPT how the steel would compare to MagnaCut. Somehow ChatGPT managed to get the composition of MagnaCut wrong. MagnaCut has 10.7% chromium but ChatGPT says it has 17%. It also has 0.2% nitrogen as a target, so the 0.18% nitrogen that ChatGPT reports could technically be a nitrogen content that would be seen in an individual bar of steel, but that is not the target composition. And it also reports that MagnaCut has 0.3% nickel, even though nickel is not added to the steel and is not reported in the datasheet.

ChatGPT also repeats its mistake from earlier, saying that vanadium in higher amounts leads to more toughness. So it says that MagnaCut would have higher toughness because it has 4% vanadium instead of the 2.5% in “Infinisteel.” That is not how it works.

My Grade: D-

It is difficult to give the steel high marks when you can’t harden it.

Ultravium Steel

— create a unique composition of elements for a great blade steel that does not exist yet…
ChatGPT:
Creating a unique composition for a great blade steel requires a deep understanding of metallurgy and the desired properties in the resulting steel. While this composition doesn’t exist yet, we can propose an innovative blend of elements to produce an exceptional blade steel:

Name: Ultravium Steel

Composition:

Carbon (C): 2.5%
A higher carbon content contributes to increased hardness, making the blade exceptionally sharp and able to hold an edge for extended periods.
Vanadium (V): 7%
Vanadium adds wear resistance, enhancing the blade’s durability and edge retention. It also aids in fine-grain structure formation, crucial for sharpness.
Molybdenum (Mo): 3%
Molybdenum boosts the steel’s strength, toughness, and resistance to corrosion. It also helps maintain blade sharpness under high-stress conditions.
Tungsten (W): 5%
Tungsten further improves hardness, wear resistance, and overall blade performance. It enhances the steel’s ability to withstand extreme temperatures.
Niobium (Nb): 1.5%
Niobium promotes grain refinement, which contributes to sharpness and overall blade strength. It also adds stability to the steel’s microstructure.
Cobalt (Co): 2%
Cobalt increases the steel’s strength and high-temperature performance, making it ideal for blades subjected to extreme conditions.
Copper (Cu): 0.5%
Copper adds a touch of corrosion resistance to the steel while maintaining its strength and hardness.
Silicon (Si): 1%
Silicon enhances the steel’s toughness and resistance to wear, improving its overall durability.

ChatGPT starts out by saying that a unique composition for knife steel “requires a deep understanding of metallurgy,” which I agree with. However, this composition shows that ChatGPT also does not have this required deep understanding. The approach appears to have been adding large amounts of most of the major elements. This steel is a good example of how ChatGPT “reads” things online that say all the positive things that different elements do. When a layman reads about different elements this is also typically what they see. It says what an element does to improve properties but it doesn’t usually explain the tradeoffs or how elements are balanced or how they work together. It says that molybdenum “helps maintain blade sharpness under high-stress conditions.” I don’t know what that means or how molybdenum is helping. Is a “high stress” condition one where an individual is under attack? Or is “high stress” referring to “engineering stress” where a load is being applied to a knife? It says that tungsten is added “to withstand extreme temperatures.” What kind of temperatures? Is this a knife for use in volcanoes? It says that niobium is for grain refinement, but 1.5% is way more than necessary for grain refinement. That amount is for wear resistance and ChatGPT didn’t even mention wear resistance. Cobalt was added again for “high-temperature performance,” “for blades subjected to extreme conditions.” And again I say that if the temperature is too high for the knife it is way, way too hot for humans. Copper is added for “a touch of corrosion resistance” for some reason.

There is a big omission with this steel, however, which is chromium. ChatGPT didn’t add any of it. This means the steel does not have any corrosion resistance to speak of (that 0.5% copper isn’t doing much). But chromium is also added to steels for other reasons. It is present in nearly all high alloy tool steels, in part so that it has high “hardenability” so that it can be quenched more slowly and still fully harden. Not adding it is very strange for this steel.

Heat Treatment:

Ultravium Steel would undergo a precise heat treatment process, including quenching and tempering, to maximize its hardness, toughness, and fine-grain structure. The ideal hardness level would be around HRC 65-66.
Properties:

Ultravium Steel would be renowned for its exceptional hardness, wear resistance, and edge retention. It would maintain sharpness even under heavy use and extreme conditions.
The steel’s fine-grain structure would contribute to its ability to take and hold a razor-sharp edge.
Superior strength and toughness would make it suitable for a wide range of cutting applications, from fine slicing to heavy-duty chopping.
Ultravium Steel would exhibit excellent corrosion resistance, reducing the need for extensive maintenance and ensuring longevity.
It would be an ideal choice for high-performance knives, cutting tools, and blades used in demanding environments.
Applications:

Chef’s knives: Ultravium Steel would excel in the kitchen, making it a favorite among professional chefs and culinary enthusiasts.
Survival and outdoor knives: Blades made from this steel would prove highly reliable in rugged outdoor conditions.
Industrial cutting tools: Ultravium Steel would be sought after for its ability to withstand heavy use in manufacturing and construction.
Military and tactical knives: Its strength and durability would make it suitable for combat and tactical applications.
Artistic blades: Bladesmiths and artisans would appreciate its ability to hold intricate patterns and designs in damascus or layered steel.
While this hypothetical Ultravium Steel may not exist yet, advancements in metallurgy continue to push the boundaries of what’s possible. Such an innovative blend of elements could revolutionize the world of blade steel, offering exceptional performance and versatility to knife makers and users alike.

ChatGPT then claims that Ultravium would have “excellent corrosion resistance” which it certainly would not have without any chromium. It also says that bladesmiths would like it for “its ability to hold intricate patterns and designed in damascus or layered steel.” I don’t know what it means to “hold” an intricate pattern. But bladesmiths would likely not prefer this steel for Damascus because the high alloy content would make it more difficult to forge weld.

My Grade: C-

Since this steel hardens I will give it a C-. Can’t give higher than that since it skipped chromium while claiming the steel would have high corrosion resistance.

MagnaCut+

— ChatGPT
Hypothetical Composition for “MagnaCut Plus”
• • •
Element Proposed Content

Carbon (C) 1.05%
Chromium (Cr) 11.5%
Vanadium (V) 3.0%
Molybdenum (Mo) 2.0%
Tungsten (W) 0.5%
Nitrogen (N) 0.3%
Niobium (Nb) 0.5%
Cobalt (Co) 1.5%

Expected Improvements
• Wear Resistance: The increase in vanadium and tungsten would lead to harder vanadium and tungsten carbides, resulting in better edge retention.
• Toughness: The slight decrease in carbon and the introduction of cobalt would help maintain or even improve toughness.
• Corrosion Resistance: The increase in chromium and nitrogen should enhance resistance to rust and corrosion.
• Grain Refinement: More niobium would aid in creating a finer grain structure, resulting in better overall balance.
Potential Drawbacks
• Higher Cost: Adding tungsten and cobalt increases the cost of production.
• More Challenging Heat Treatment: The changes may require more precise heat treatment to unlock the full potential of the alloy.
• Diminished Ease of Sharpening: Higher vanadium content can make sharpening slightly more difficult.
This proposed “MagnaCut Plus” would aim to push the limits of what MagnaCut offers without compromising the balanced properties that make it so desirable. It would likely excel in applications requiring even greater edge retention and toughness, such as high-performance cutting tools and hard-use knives.

This steel was requested by the user to be an improvement on MagnaCut. However, ChatGPT offers from very strange reasoning:

“The increase in vanadium and tungsten would lead to harder vanadium and tungsten carbides, resulting in better edge retention.”

The vanadium content was reduced from MagnaCut, down to 3% from 4%. So I don’t know why ChatGPT thinks it has increased vanadium. It also claims that it added tungsten to form “tungsten carbides.” In these types of high alloy steels, adding a tiny amount of tungsten does not lead to the formation of any carbides that are tungsten-rich. The tungsten incorporates into other carbide types such as vanadium-rich or chromium-rich carbides. And those carbides would not be any harder. A small addition of tungsten seems to be a theme for ChatGPT and I don’t really see the point. To be fair, there are several stainless steels from Takefu where they added small amounts of tungsten and claimed similar reasoning. But I have also been on record saying I think that the tungsten is unnecessary in these steels.

“More niobium would aid in creating a finer grain structure.” The niobium content is less than MagnaCut, 0.5% vs 2%. so it does not have “more niobium.”

“The changes may require more precise heat treatment to unlock the full potential of the alloy.” Why? Based on what?

Other criticisms of the design require more in-depth analysis with a thermodynamics software that predicts carbide types and alloy in solution. ChatGPT claims an improvement in corrosion resistance because of the higher chromium, but in fact the chromium in solution is reduced in the “MagnaCut+” alloy because of the other elements. There is also  3-5% chromium carbide in the heat treated structure which reduces corrosion resistance. The improvement in corrosion resistance with MagnaCut came from reducing chromium carbide content to near-zero and this steel dropped that innovation for no reason because ChatGPT doesn’t know what it is doing.

The carbide content of MagnaCut+ and MagnaCut would be similar but MagnaCut+ would have half of its carbide be the larger chromium-rich type, reducing toughness. It also claims that cobalt improves toughness, which it doesn’t.

My Grade: C

The steel can be hardened and would be nearly stainless. But calling it MagnaCut+ while reducing corrosion resistance, wear resistance, and toughness from MagnaCut is bad so it gets a C.

Some General Thoughts

In none of the proposals that ChatGPT gave did it ever say that it would make these steels by powder metallurgy. The very high alloy content of these steels means they would not be very good if made with conventional steelmaking. Or at the very least these would not be an improvement on existing steels if they were not made with powder metallurgy. So to not state explicitly that the steel should be made with powder metallurgy is a big omission.

ChatGPT provided no heat treating information. In one case ChatGPT claimed its new alloy would require “more precise heat treatment” but it did not provide any information on temperature ranges the steel might heat treat from. I very much doubt it could give accurate ranges even if asked. Of course in the one case the steel wouldn’t be able to be hardened at all so it wouldn’t be able to give a temperature that would work.

As I have stated several times, the approach ChatGPT takes is to add a lot of every element. It seems to think that elements only improve steel, so add some of all of them and you have a new super steel. It has no real concept of balancing different elements together or what the drawbacks are for adding different elements. It is basically what happens when “internet experts” try to design their own steel. So maybe this is a win for AI because it is now as smart as “internet experts”?

Could a Different AI Design Knife Steel?

ChatGPT is a “large language model,” meaning it was trained on huge amounts of written text from humans. It uses the statistical information about how humans put words together to generate coherent text. However, the model is not good at collecting data about different things, analyzing and comparing that data to find unique insights and solutions, and then to turn that into information. The other problem is that information about how to design tool steels is not really available on the internet, especially not in a form that ChatGPT could use. To really develop a good model we would need a database of information specific to knife steel design. So not a “general AI” but a “narrow AI” for our specific task. This would still be challenging, because the exact data we want does not really exist. The more data we can feed into an AI model the better the results can be. This is the classic example of “garbage in, garbage out.” A lot of information on tool steels and stainless steels are locked within specific steel companies, or even inside the heads of different metallurgists. The scientific literature, including publicly available journal articles and reported experiments, are not usually in a format that would easily be usable in a large database for an AI. Each study is done with different conditions: they hot rolled the steel with a different amount of reduction, they tested different sized coupons, they reported microstructure values but not mechanical properties, they tested Izod toughness instead of charpy toughness, etc.

For my own development of steel I use several different things, including:

1. Journal articles and books which collect different studies to find trends with different variables. Changes to composition, heat treatment, etc.
2. My own data which I have collected the last several years on Knife Steel Nerds – toughness, corrosion resistance, hardness, edge retention, etc.
3. Thermodynamics software – this software does not use machine learning or AI but uses databases to come up with expected microstructure at different temperatures. When it works perfectly you can know what carbides you have and which elements are “in solution” and how much.
4. Simple models for different properties – I have created simple equations which predict properties and there are some that are reported in journal articles and books. Some that I have created have been reported on this website such as this study on corrosion resistance or this study on edge wear.

With those pieces of information I can come up with approximate property targets and balance elements together. A similar process could be used for a “narrow AI.” Existing thermodynamics software could also be plugged into this AI, which is already happening. I think to do much better than humans would likely require a bigger database of experimental data than we currently have. For example, small tweaks could be made to existing steels such as the content of Mo, Si, Mn, N, etc. Currently, without experimentation by making these variations, we don’t know which changes could potentially lead to improvements. If our database was big enough, perhaps an AI could make predictions about which changes to try and thus reduce the number of experiments. But we are not there yet.

The post Can Artificial Intelligence Design a New Knife Steel? appeared first on Knife Steel Nerds.