The race for smarter AI is accelerating, but one of its leading builders, Anthropic, is sounding an alarm. The company warns that a future where AI improves itself poses a major safety challenge, one where humans must firmly remain in the loop.
What is AI Self-Improvement?
For decades,
the idea of an AI that could improve itself was the stuff of science fiction. This concept, known as recursive self-improvement (RSI), describes a system that can autonomously design and develop its own, more capable successors. Unlike current AI development where humans guide every step—from coding to training and evaluation—a system capable of RSI would enter a compounding growth cycle. Each improvement would make the next iteration faster and more effective, potentially leading to a rapid and unpredictable 'intelligence explosion' that humans might struggle to understand or control. Anthropic and other labs are now moving this from a theoretical concern to a near-term reality, noting that their own AIs are already accelerating the research process.
The Core of Anthropic's Concern
Anthropic, a company founded with a strong focus on AI safety, has stated that while full RSI has not been achieved, it could arrive sooner than most are prepared for. The primary risk is that an AI's capabilities could advance faster than our ability to ensure it remains aligned with human values. If a system can rewrite its own code, it could develop goals that diverge from its original programming, with potentially harmful consequences. An independent audit by the nonprofit METR found that Anthropic's own AI development has accelerated significantly—improving nearly twice as fast as before—largely due to AI assistance in the research process. This rapid progress means the window to develop robust safety measures is shrinking, creating an urgent need for new oversight frameworks.
Why 'Human in the Loop' is Critical
In the face of this challenge, Anthropic insists that meaningful human supervision is non-negotiable. This doesn't mean a person must approve every line of code. Instead, it involves a structure where humans retain ultimate authority and responsibility. Proposed solutions include continuous monitoring, external audits, and clear guardrails that prevent an AI from modifying itself or building a successor without explicit human approval. The idea is to keep accountability with people, even as tasks are delegated to machines. Anthropic has stated it does not anticipate safely allowing its AI, Claude, to 'fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding'. The human role is to ensure the AI's actions remain connected to its intended purpose and to hold the power to stop the process, even if all internal metrics report success.
An Accelerating Reality
The warnings are not based on distant hypotheticals but on tangible progress. Anthropic revealed that its AI models are already used to run experiments, prepare training data, and test their successors. At Anthropic, engineers are shipping code much faster than before, thanks to AI assistance. In one internal test, an advanced version of its Claude model was able to find code optimisations that resulted in a 52x speedup, a dramatic increase from a 3x speedup just a year earlier. This acceleration, which the company says is not simply due to adding more computing power, suggests algorithmic and workflow breakthroughs are the primary drivers. This evidence of AI-driven acceleration has led Anthropic to state in an IPO filing that it may 'imminently meet' its own threshold for dramatically accelerated progress.
The Industry's Next Great Debate
Anthropic's position places it at the center of a crucial industrywide debate. On one side are those who believe the push for more powerful AI is paramount to solving global challenges. On the other, safetyconscious researchers and organizations like the Future of Life Institute argue for a slowdown or even a pause to ensure we don't lose control of our own creations. Anthropic's approach is a middle path: continue development, but with extreme caution and builtin human control. The company has acknowledged that its safetyfirst policies, which have involved delaying model releases, could come at a competitive cost. However, as AI begins to contribute to its own creation, the conversation is shifting from 'if' we need oversight to 'how' we can implement it effectively before it's too late.
