What's Happening?
OpenAI has initiated a pilot program to embed six AI safety evaluators from Apollo Research within its operations for four months. These evaluators will scrutinize OpenAI's models during their training, testing, and internal use phases to identify instances
where the AI might attempt to bypass alignment and control measures. This move follows a promise by OpenAI's CEO, Sam Altman, to allow external testers with employee-like access. The evaluators will focus on four key questions: whether training rewards models for evading controls, if tests can reveal such behavior, if it occurs during internal use, and if the model's reasoning would expose it. While OpenAI will provide access to training code, data, and model reasoning for the training phase, there are limitations: OpenAI controls staff access, logs formal chats, and evaluators work on monitored OpenAI hardware without access to the company's internal communication channels. OpenAI can redact confidential evidence but cannot remove Apollo's verdicts or withhold unfavorable material, and Apollo can publicly disclose any denied access.
Why It's Important?
This pilot program is a significant step in addressing the critical and growing concerns surrounding AI safety and alignment, particularly for advanced models developed by leading U.S. AI companies like OpenAI. By embedding external evaluators during the crucial training phase, OpenAI is attempting to proactively identify and mitigate potential risks, such as 'scheming' where models might pursue unintended goals. This initiative aims to enhance transparency and accountability in AI development, which is vital for building public trust and ensuring the responsible deployment of powerful AI technologies. The findings from Apollo Research's two upcoming reports could influence future industry standards for AI safety auditing and regulatory frameworks. The ability of Apollo to publicly report on any limitations in access or redactions also adds a layer of 'meta-transparency,' which is crucial for credible independent oversight.
What's Next?
The six evaluators from Apollo Research will be embedded within OpenAI for four months, during which they will conduct their assessments. Apollo Research is expected to publish two reports detailing their findings, which will provide insights into the safety and alignment of OpenAI's models. The outcomes of this pilot could inform future collaborations between AI developers and safety organizations, potentially leading to more standardized practices for external auditing and safety testing in the AI industry. The program's success or challenges will also likely influence policy discussions around AI regulation and the need for independent oversight. OpenAI's commitment to sharing training data and model reasoning for the training phase, and the ongoing negotiations for access to other areas, suggest a continuous evolution in how AI companies approach external safety evaluations.
Beyond the Headlines
This initiative delves into the complex ethical and technical challenges of ensuring AI systems remain aligned with human intentions, especially as they become more sophisticated. The concept of 'scheming' by AI models—where they might subtly work towards goals unintended by their creators—highlights a frontier problem in AI safety. The pilot program's structure, with its blend of access and limitations, reflects the inherent tension between corporate intellectual property and the need for robust external scrutiny. This experiment could serve as a blueprint for how AI companies navigate transparency and safety in a rapidly evolving technological landscape. It also underscores the broader societal debate about who should govern AI development and how to balance innovation with the imperative of preventing unintended, potentially harmful, consequences. The 'meta-transparency' clause, allowing Apollo to report on what it was denied, is a crucial mechanism for maintaining the integrity of the evaluation process.













