In a striking move, a leading AI company is sounding the alarm about the very technology it's building. Anthropic, creator of the Claude AI models, has warned that progress is happening so fast it could outrun our ability to control it.
The Core of the Concern
In a recent filing
with the U.S. Securities and Exchange Commission, Anthropic stated that artificial intelligence could soon accelerate the development of its own successor systems at a pace that outstrips human safety safeguards. This concept, known as 'recursive self-improvement', is a long-theorized milestone where AI becomes so capable that it can significantly speed up the creation of the next, more powerful generation of AI. The company’s internal estimates suggest it may soon reach a threshold for 'dramatically accelerated progress attributable to AI'. The central fear is not that AI will suddenly become malevolent, but that its capabilities will advance faster than our ability to develop, test, and implement the necessary safety protocols to keep it aligned with human intentions.
AI Building AI Is Already Here
This isn't a distant, theoretical future. Anthropic disclosed that its AI model, Claude, is already deeply integrated into the company's own research and development. In September 2026, models reportedly wrote over 90% of the company's codebase contributions. AI now assists with creating training data, running experiments, and even conducting safety audits. Anthropic's own analysis found that the share of R&D tasks where AI performs most of the work under human supervision surged from just 1% to 26% in a single year. This has led to massive productivity gains, with engineers producing significantly more code than before. While this speeds up progress, it also means the development loop is closing, bringing the risks of outpacing safety measures from a future possibility into a present-day concern.
Why Human Supervision is Non-Negotiable
Given these accelerating capabilities, Anthropic has been clear: it does not anticipate being able to safely allow its AI to 'fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding'. This isn't just about having a person in the room to unplug the machine. It means maintaining robust human oversight, judgment, and control throughout the development pipeline. As AI systems take on more complex tasks, the role of human engineers is shifting from writing code line-by-line to directing and reviewing AI-generated work. Anthropic is so committed to this principle that it is updating its own 'Responsible Scaling Policy' to ensure human control is preserved, even if it means slowing down its own progress compared to competitors.
A Race Against Ourselves
Anthropic’s public warnings highlight a fundamental tension in the AI industry. Companies are in a fierce race to build the most capable and profitable models, creating immense pressure to move fast. However, the critical work of safety research, validation, and alignment is inherently slow and deliberate. This has led Anthropic to take precautionary measures, such as delaying the release of certain powerful models and restricting access to them. The company's 'Responsible Scaling Policy' is designed as a framework to tie the scaling of AI capabilities to the implementation of stronger safeguards, similar to how biosafety levels work. The challenge, as the company acknowledges, is that if one developer pauses for safety while others race ahead, the entire ecosystem could become less safe.
