June 4 (Reuters) - Anthropic said on Thursday frontier AI developers should establish a coordinated, verifiable way to slow down or temporarily pause development if advanced systems begin improving themselves faster than society can manage the risks.
AI that can build itself would be a major development in the history of technology, but "full recursive self-improvement also might increase the risks of humans losing control over AI systems," the AI startup said.
"If systems are capable of fully building
their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important."
As an example, Anthropic said that as of May, more than 80% of the code merged into its codebase was authored by Claude.
It would be "good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology," the company said.
However, it cautioned that unilateral or poorly coordinated slowdowns could backfire if less cautious actors continue advancing, potentially reducing overall safety.
It highlighted that a meaningful pause would require agreement among "multiple well-resourced labs" operating at the technological frontier, as well as rules on what conditions would trigger or lift such a pause and who would oversee it.
A unilateral pause by a single company would be easier to implement, Anthropic added, but would have limited impact, primarily shifting leadership rather than fostering broader global deliberation.
Its research arm, Anthropic Institute, plans to study and help build systems that would be necessary to support a slowdown.
In the coming months, Anthropic plans to convene discussions involving policymakers, researchers, civil society groups and other AI firms to examine key questions.
These questions include how to manage AI-related risks such as recursive self-improvement and how to improve mechanisms for coordination.
Last month, Anthropic concluded a fundraising round that valued the company at $965 billion and confidentially filed for a U.S. initial public offering on Monday.
(Reporting by Juby Babu in Mexico City; Editing by Shreya Biswas)
