What's Happening?
The Brookings Institution has released a comparative analysis titled "How are countries evaluating AI models?", which examines national AI model evaluation programs. The report indicates that countries such as Singapore, Japan, and the UK are moving beyond
basic safety checklists to incorporate more sophisticated evaluation criteria. These expanded programs now address multilingual safety differences, the specific risks associated with autonomous AI agents (agentic risk), sensitive information leakage, the potential for fraud enablement, and cybersecurity capabilities. Microsoft is noted as a collaborative participant in these cross-government evaluation efforts. The research draws upon established frameworks, including voluntary standards from the OECD and ISO, alongside various national government programs. The report underscores that the scope and methodology of AI evaluation differ significantly across jurisdictions, reflecting a global trend where regulators and courts are increasingly demanding demonstrable proof of AI vendor testing rather than mere assertions.
Why It's Important?
This development is crucial for the U.S. technology sector and regulatory landscape as it signals a global shift towards more rigorous and comprehensive AI governance. For U.S. companies developing or deploying AI, particularly those operating internationally, understanding these evolving evaluation standards is paramount. The focus on 'agentic risk' means that AI systems capable of autonomous action will face heightened scrutiny, potentially impacting the design and deployment of advanced AI applications across various industries. Furthermore, the emphasis on multilingual safety, data leakage, fraud, and cybersecurity highlights growing concerns about the societal and economic impacts of AI. U.S. policymakers and regulatory bodies may look to these international precedents when formulating domestic AI evaluation frameworks, influencing future compliance requirements and potentially shaping the competitive landscape for AI innovation.
What's Next?
Enterprises utilizing AI are advised to proactively engage with their AI vendors to ascertain which national government evaluation programs their models have undergone, the specifics of the testing, and the results. Companies should also update their AI vendor due diligence checklists to ensure they cover agentic risk, multilingual behavioral differences, and sensitive data leakage, moving beyond general safety benchmarks. It is recommended that organizations map their AI deployments against the jurisdictions mentioned in the Brookings report (Singapore, Japan, UK) to identify any models that have not been evaluated under these programs. Additionally, updating model deployment gate processes to require vendors to disclose any changes in their participation or results from national evaluation programs before approving model updates will be a critical next step for ensuring compliance and mitigating risks.
Beyond the Headlines
The expansion of AI evaluation programs into areas like agentic risk and multilingual safety differences points to a deeper recognition of AI's complex and multifaceted impact on society. Beyond technical compliance, these evaluations touch upon ethical considerations related to AI autonomy and potential biases embedded in language models. The involvement of major tech companies like Microsoft in cross-government evaluation work suggests a growing understanding that AI governance requires collaborative efforts between the public and private sectors. This trend could lead to the development of international norms and standards for AI, fostering greater trust in AI systems but also potentially creating new barriers to entry for companies unable to meet stringent global requirements. The long-term implications include a more responsible and secure AI ecosystem, but also the challenge of harmonizing diverse national approaches to AI regulation.
