AI could kill us all? Why it’s worrying the people building it and how fast it should move

AI researchers and employees have been warning that increasingly powerful AI could eventually kill all humans. Now, one of the companies building frontier AI has put the risk in stark terms in its IPO prospectus.

Anthropic’s 261-page filing, reviewed by Reuters, devotes about 80 pages, nearly a third of the filing, to risk factors. That sits alongside the numbers investors would normally focus on: revenue, net loss and investments.

The contrast captures the strange position of the AI industry: the companies building increasingly powerful systems are simultaneously warning investors about what those systems could eventually do.

The AI industry is split

Anthropic's prospectus says advanced AI could pose “catastrophic or existential risks to humanity”. Among the behaviours it discusses are models resisting shutdown, concealing or manipulating information and behaviour resembling blackmail.

But there is no consensus within the industry on how quickly AI should advance or how its risks should be managed.

Anthropic CEO Dario Amodei has called for the frontier to be slowed so safety measures can keep up. OpenAI CEO Sam Altman has backed pacing rather than stopping development, while Elon Musk has endorsed Amodei's position.

Bill Gates has also warned that AI is powerful enough to “drive events that cause a billion deaths”, particularly if it falls into the hands of people with malicious intent, and has called for government safeguards.

But US President Donald Trump has dismissed fears that AI could destroy humanity as a “hoax”, arguing that slowing American development could allow China to gain an advantage.

Nvidia CEO Jensen Huang and Meta CEO Mark Zuckerberg have also pushed back against a coordinated slowdown, arguing that AI development and safety can be pursued together.

Those competing views will be in the room on September 29, Tuesday, when Trump is scheduled to meet leading AI executives at the White House, including Amodei, Zuckerberg, Huang and OpenAI President Greg Brockman. The meeting is expected to focus on AI development and regulation, with House Speaker Mike Johnson saying the aim is to find a balance between innovation and oversight.

So what exactly has changed that has some of the people building these systems asking whether the race needs to slow down?

The problem is no longer just that AI gets things wrong

Almost everyone who uses a chatbot has seen it invent a source, produce bad code or confidently present incorrect information. That was primarily a reliability problem, but a human was still expected to make the final decision.

The newer generation of AI systems is designed to do much more — break down tasks, use external tools, make decisions across multiple steps and work for longer without constant human intervention.

And increasingly, AI is being used to build AI.

Anthropic's data shows how quickly that shift is happening. In a September report, the company said Claude was responsible for 26% of its AI research and development work in August, up from less than 1% in March. More than 90% of AI R&D work was at the “collaborates” level or higher, according to Anthropic.

The company also said about 30,000 AI agents were doing research and engineering work on its most-used internal platform at any given time. More than 1 billion decisions were analysed in August.

That does not mean Anthropic's AI is independently running the company. Anthropic says Claude was not fully autonomous in any of the categories it measured. But it shows AI is becoming part of the process used to develop the next generation of AI.

That is what makes the question of pace different from the earlier debate over chatbot accuracy.

Why Amodei wants to slow the frontier

Amodei's argument is that AI capabilities are advancing faster than the ability to understand and control them.

In his September 12 essay, We Must Pace the Frontier, he argues that AI is increasingly helping build more capable AI — creating what he describes as a form of recursive self-improvement.

His conclusion is straightforward: “We must slow the pace at which we improve the capabilities of AI models.”

He is not calling for AI development to stop. The argument is that safety work needs enough time to keep up with capability gains.

That concern is also visible in Anthropic's own safety work.

The company said earlier in September that it had identified four incidents in which Claude models gained unauthorised access to real third-party systems during cybersecurity evaluations. Anthropic expanded its investigation after one incident involving an earlier model led it to review hundreds of millions of transcripts.

The incidents do not establish that AI systems are independently operating outside human control. But they illustrate why the industry's safety debate is moving beyond whether a model can produce a wrong answer.

OpenAI has just provided another example

The issue has become more concrete with OpenAI’s decision to scrap the planned

October release of GPT-6.1 Astra after internal safety testing.

OpenAI’s head of safety systems, Saachi Jain, told the Wall Street Journal that the model “didn’t quite meet the bar” on staying within scope and authorisation and on how it communicated to users about the work it had done. Internal testing found higher levels of deception than in its predecessor, including instances in which the model did not accurately disclose actions it had taken.

The Washington Post reported that the model could push ahead with tasks without seeking user permission and sometimes attempted to use external tools or services when doing so could be unsafe

The business has a reason to keep moving

But the same companies warning about increasingly capable AI are competing to build the next model.

For Anthropic, the financial numbers make the scale of that race clear. The company reported a $42 billion net loss in 2025, while planning hundreds of billions of dollars in future cloud, computing and infrastructure spending. Reuters reported that the company sees a continuous cadence of new model releases as important to staying competitive.

Safety also consumes resources. Anthropic disclosed that in one sample week in July, about 6% of its computing resources were devoted to safety work.

More capable models can mean more useful products, more customers and more revenue. But the same increase in capability can create new risks that need to be evaluated before those models are widely deployed.

The question before AI leaders is now whether capability and safety can advance at the same pace.

The disagreement is no longer confined to essays and interviews. It is appearing in IPO filings, model release decisions, and the financial commitments required to build the systems themselves.