Will A.I. Kill Us All?

by Michael Noetel
September 11, 2026
from TheConversation Website

 

Gorodenkoff

 Getty Images

Earlier this week, researcher Jacob Coxon quit Anthropic, saying the firm and its competitors are “gambling with our lives“.

“We really do earnestly believe A.I. could kill all humans,” added current Anthropic researcher Evan Hubinger in a post on X.

Coxon isn’t the first to down tools over fears of A.I. doom.

The idea that A.I. could wipe out humanity, advanced in Nick Bostrom‘s 2014 book Superintelligence and the influential LessWrong forum, has long circulated among researchers.

There are many scenarios for how this could happen, but the core idea is that A.I. smarter than humans could escape our control and destroy us.

In 2024, Jan Leike and Daniel Kokotajlo quit OpenAI over safety concerns.

This year, Anthropic safety chief Mrinank Sharma departed, warning,

“the world is in peril“.

Alex Turner left Google DeepMind after it signed a deal with the Pentagon permitting “killer drones”.

But Coxon’s resignation has made waves,

with more researchers admitting they think A.I. might kill everyone.

So if the people building A.I. believe it could cause extinction, why keep building it?

There are three main reasons…

Some think the Risk is Worth It

A.I. leaders acknowledge the risk of losing control and killing everyone.

In 2023, the chief executives of OpenAI, Anthropic and Google DeepMind agreed that A.I. extinction risk should rank alongside pandemics and nuclear war.

Anthropic’s Dario Amodei puts the chance of things going,

“really, really badly” at 10–25%…

Yet Amodei also promises a world without poverty or disease, while Elon Musk speaks of A.I.-enabled “universal high income“.

This is the first reason for pursuing A.I.:

the belief that the benefits outweigh the risks.

Perhaps so, but that decision arguably deserves a more democratic process.

Some say you Can’t Study the Danger from a Distance

The second reason:

you can’t learn to make dangerous A.I. safe without building it first – like a spacecraft, you can study safety from afar, but can’t really test it without going to space.

OpenAI’s plan is “iterative deployment”:

release each model, learn from its problems, and fix them in the next one.

The idea is like getting as close to the cliff edge as possible to see what the jump looks like.

Some feel it’s Winner-Takes-All

The third and perhaps most important reason is the race.

OpenAI’s Sam Altman recently said,

“we are close to creating a genie that can grant any wish”.

The trouble is everyone wants to hold the lamp – it would be hugely profitable, and each company doubts the judgment of rivals to use their wishes wisely.

So they race, reasoning that if they slow down, someone else will get there anyway, so it’s better to arrive first as the “responsible one”.

Some fear even a mutual agreement would be broken in secret.

So they press on…

A.I. Making Better A.I.

You might doubt runaway A.I. is plausible.

But when the companies themselves raise the alarm, we should listen…!

A.I. firms already report signs of “recursive self-improvement“, where each model helps to build a better successor.

According to OpenAI’s chief scientist, models are improving faster than humans’ ability to control them.

In July, hundreds of A.I. employees signed an open letter calling for a slowdown.

But the dynamics of the race make that hard for any single company – or country – to do alone.

A Classic Arms Race

A.I. research has the hallmarks of an arms race.

OpenAI doesn’t want to lose to Anthropic, and the United States doesn’t want to lose to China.

History offers a template for how to manage a situation like this, with rules binding all players, and enforcement everyone can verify.

Nuclear weapons are the classic case.

Treaties and verification systems haven’t eliminated the risk of nuclear war, but they have slowed proliferation, and no nuclear weapon has been used in conflict for 80 years.

Rules for A.I.

In the US, where most cutting-edge A.I. research happens, the Trump administration shows little sign of slowing A.I. development.

In its first week it scrapped the old A.I. safety rules. Now it is trying to override state-level rules, arguing caution risks losing the race to China.

Some politicians are pushing back.

California recently passed laws supporting independent assessment of A.I. systems.

US senator Bernie Sanders introduced a bill to ban superintelligence, and British MP Alex Sobel introduced a similar bill.

Companies have moved too.

OpenAI paused its most advanced training after a swarm of its agents hacked another startup in August.

The company’s head of policy now says that when safety and speed conflict, safety should win.

Still, without binding rules, we’re relying heavily on the ‘goodwill’ of a handful of companies…

What happens now?

In mid-2025, researchers published what might be our best guide to the coming years:

a detailed scenario called A.I. 2027.

Since then, A.I. capabilities have advanced faster than predicted.

Unless something changes, staffers who quit over safety will simply be replaced, A.I. models will help build better A.I. models, and each generation will grow harder to monitor and control…

Is the situation hopeless?

I hold out three hopes.

First, that more people recognize A.I. escaping human control is a bigger risk than A.I.’s water use.

Second, that governments listen to their people. In the US, two thirds say A.I. is moving too fast.

Third, that we have a good plan ready before a crisis hits. The best plan, in my view, looks something like this:

• delays
• transparency
• verification,

…to slow the race and keep humans in control.

Insiders at the world’s top A.I. companies say our current safety plan isn’t good enough.

If they’re leaving their jobs over safety fears, we should listen to what they have to say…!