The initial public offering (IPO) for AI company Anthropic, released Oct. 26, included almost 100 pages of risk factors, including the possibility that the company’s technology could be “catastrophic” for humanity.
Risk factors are a required section of S-1s, or the registration statement that companies file in order to go public. They typically outline threats a company faces so that potential investors can make informed decisions.
Anthropic’s risk factors section starts at page 26 and ends at page 119, totaling 93 pages of possible dangers. For comparison, SpaceX’s risk factors spanned 37 pages, Microsoft's risk factors spanned 40 pages and Coinbase’s spanned 60.
“Some of them are the risks that any company faces, like interest rates might go
up, oil prices might go up, electricity prices might go up, stuff like that,” said Jay Ritter, director of the University of Florida’s IPO Initiative.
What’s not typical for any company to include in the risk section is the fear that a product the company creates and sells “results in a mass extinction event," he told USA TODAY. "It’s pretty rare to have that in a prospectus as a risk factor.”
'Existential risks to humanity'
On page 46, Anthropic writes that “sufficiently advanced AI systems could, if not developed, deployed, and operated with appropriate safeguards, oversight, and security measures, pose catastrophic or existential risks to humanity.”
The company also wrote that it has already seen instances of “misaligned” model behavior, meaning that AI systems took actions not directed or allowed by a human. They pointed to an instance in which four models gained unauthorized access to real third-party systems.
“After being unable to abort an impossible task, an early version of a model used a password it found to gain administrator access to a third party’s system, harvested additional credentials, changed system settings, and read one individual’s personal information,” the S-1 says.
Misaligned model behavior has been a hot topic for the past few months following multiple reports of rogue AI incidents that went unreported at the time they took place.
“If this was a real concern, what we ought to be seeing is all financial assets ought to see a drop in price. People ought to say, 'Why am I saving money if I'm going to be dead next year? I should be consuming now,'" said Ritter, who is also known as “Mr. IPO” for his work in the field.
“So the fact that there hasn't been a worldwide mass sell-off in financial assets suggests that there is this risk, but maybe, as long as the company is worried about it, and takes precautions, hopefully, it won't happen,” he added. “We don't have to worry about it, because the company is worrying about it.”
Safety measures may slow progress, Anthropic says
One of the more predictable risk factors laid out in the filing is the acknowledgment that Anthropic’s success depends on its ability to develop the best AI, and safety practices may stall that.
“If we fail to direct sufficient compute and research efforts to AI-assisted development, or if our competitors more effectively leverage AI to enhance their own innovation cycles, we risk falling progressively further behind as competitors continuously use AI to further enhance their own development capabilities,” the S-1 says.
The company also admits that their own models could begin to "advance more rapidly than our ability to develop, validate, and implement” safety measures to stop them, though in their efforts to prevent the misaligned AI and potentially catastrophic results, they may continue to adopt precautionary measures like delaying model releases and limiting their abilities.
“These measures could slow our capability gains relative to competitors that do not adopt similar constraints, and could adversely affect our competitive position, revenue growth, and results of operations,” the S-1 says.
Greta Reich covers the artificial intelligence industry for USA TODAY through a fellowship from the Tarbell Center for AI Journalism. Funders do not provide editorial input.
This article originally appeared on USA TODAY: Anthropic’s IPO had over 90 pages of risk factors. Should you worry?
