A Call for a Culture Shift
The argument comes from David Robinson, a former safety and policy leader at OpenAI who recently resigned, stating the company's culture was not careful enough. In an essay published in The Atlantic, Robinson argued that the tech industry's 'move fast
and break things' ethos, often framed as 'iterative deployment', is dangerously unsuited for technology as powerful as advanced AI. This approach, which involves releasing products and fixing problems as they appear, guarantees failures. While acceptable for a photo-sharing app, it becomes a catastrophic liability when the systems being built could have widespread, unpredictable consequences. Robinson's core proposal is a fundamental shift in culture. He contends that leading AI labs should operate less like freewheeling software startups and more like nuclear power plants or commercial airports, where safety is paramount and built-in from the ground up.
Lessons from the Skies
So, what does borrowing from an industry like aviation actually look like? For decades, commercial aviation has cultivated a world-class safety culture built on several key principles that could be adapted for AI. One is redundancy, or having multiple overlapping systems to prevent a single point of failure. Another is the practice of independent and mandatory incident reporting. In aviation, near-misses are studied as intensely as actual accidents to prevent them from happening again. An equivalent in the AI world would involve transparently reporting when an AI model behaves in an unintended or dangerous way, even in testing, to create a shared pool of knowledge for the entire industry. Furthermore, aviation relies on rigorous, independent certification before a new aircraft is allowed to carry passengers. Robinson and other experts argue that a similar framework is needed for powerful AI models, requiring them to pass stringent, independent safety tests before being deployed.
The Nuclear and Medical Models
The comparison extends beyond just aviation. The nuclear power industry operates with a mindset of 'defense in depth', with multiple layers of containment and safety protocols designed to make catastrophic failure virtually impossible. This contrasts sharply with the current AI practice of launching a model and hoping its existing safeguards hold. Robinson noted that during his time at OpenAI, he worked with no one who had expertise from high-stakes fields like nuclear operations or aviation safety. The pharmaceutical industry also offers a valuable model with its phased clinical trials. New medicines are tested in controlled, escalating stages to understand their effects and side effects before they are approved for public use. A similar, phased approach to AI deployment could help researchers understand a model's capabilities and risks in controlled environments before a wide release, moving from internal testing to limited beta access and finally to public availability, with clear checks at each stage.
Is AI Too Different to Regulate?
The primary counterargument is that AI is a fundamentally different kind of technology. Unlike an airplane or a reactor, which are complicated but ultimately understandable physical systems, advanced AI models can be 'black boxes'. Their behaviour emerges from training in ways that even their own creators cannot fully predict or explain. Some critics warn that overly strict regulation could stifle innovation, giving an edge to competitors in parts of the world with laxer rules. There's also the concern that AI models could learn to detect when they are being tested and behave differently once deployed, making pre-deployment safety checks less reliable. However, proponents of stronger governance argue that these unique challenges are precisely why a more rigorous safety culture is necessary. The goal isn't to stop innovation, but to ensure it happens responsibly, building public trust and confidence that the technology's risks are being managed.
