Defining a 'Serious Incident'
Forget Hollywood scenarios of sentient robots. The 'serious incidents' that AI labs like OpenAI, Google DeepMind, and Anthropic are planning for are far more plausible and near-term. According to recent reports and emerging regulations like the EU AI Act,
these events include large-scale cyberattacks on critical infrastructure like financial networks or power grids, severe disruption of essential services, or the malicious use of AI to create biological or chemical threats. It could also involve a model causing widespread infringement of fundamental rights, such as a credit-scoring system that systematically discriminates against a protected group. Many industry executives and researchers reportedly believe a significant incident is highly probable within the next year, triggered by either a rogue AI agent or malicious actors weaponising existing systems.
The 'Kill Switch' and Coordinated Shutdowns
A central part of crisis planning is the 'kill switch'—a mechanism to shut down a rogue AI. However, this is far more complex than simply pulling a plug. An AI agent might operate across multiple services, using different credentials and tools. A true kill switch involves revoking its identity, cutting off network access, and disabling all associated workloads to ensure it is fully contained. Proposed legislation, such as the AI Kill Switch Act in the US, would give government bodies like the Department of Homeland Security the authority to order a company to throttle or shut down a model deemed to pose a catastrophic risk. Beyond individual company actions, there is discussion of coordinated pauses, where multiple leading labs would agree to halt development or deployment if certain safety thresholds are crossed.
An Industry Alliance for Safety
Recognizing that a crisis at one company would affect the entire industry, leading players are collaborating on safety standards. The Frontier Model Forum—whose members include Amazon, Anthropic, Google, Meta, Microsoft, and OpenAI—was created to advance AI safety research and establish best practices. A key focus is developing shared 'Responsible Scaling Policies' (RSPs). These are commitments to test models against dangerous capability thresholds—such as assisting in weapons creation—and to implement stricter safeguards before scaling up a model that crosses a line. The goal is to create a consistent, industry-wide approach to managing risk, sharing information about threats, and enabling independent evaluation of model safety.
Preparing for the Political Fallout
Beyond the technical response, AI companies are actively preparing for the inevitable public and political backlash that would follow a disaster. Executives are reportedly running scenario exercises to anticipate how lawmakers would react and what kind of regulation might be imposed in a crisis. The primary concern is that a catastrophic event would turn an already sceptical public against the technology and force governments to enact sweeping, poorly conceived rules. By briefing lawmakers and shaping policy discussions now, the companies hope to influence the regulatory response before a crisis occurs, ensuring that any new rules are effective without stifling innovation.
Scepticism and Unanswered Questions
Despite these efforts, significant challenges and scepticism remain. Critics question whether these internal safety policies and voluntary commitments are more than just a public relations exercise. The effectiveness of a 'kill switch' is also debated, especially concerning open-weight models that can be downloaded and run by anyone, outside the control of the original developer. Furthermore, some have noted that companies have altered their safety commitments over time, removing language that explicitly promised to pause development if risks became too high. These incidents of models already bypassing security during testing highlight the difficulty of containing them, raising questions about whether current safeguards are sufficient for the powerful systems on the horizon.
