AI risks are growing – and, increasingly, they're emerging once algorithms have passed laboratory safety tests. Managing this risk can only be done with cooperation.
Who is responsible when an artificial intelligence (AI) system passes laboratory tests but causes harm in the real world?
AI is one of the most powerful technologies ever deployed at scale before its long-term consequences are fully understood. It presents a distinctive challenge because new capabilities and failure modes can continue to emerge after deployment as autonomous systems interact with real-world data, tools, users and operational systems at a speed and scale that make their consequences difficult to assess in advance.
As AI incidents and near misses multiply, reporting and shared learnings have not kept pace. Industry leaders, policymakers and scientists have called for measures ranging from pacing frontier AI development to strengthening independent evaluations and international cooperation.
This raises a question: what should governments, developers, integrators and deployers do now, while AI capabilities and their risks are still evolving?
Much of the safety debate has focused on the companies developing frontier AI models. Yet the consequences of AI increasingly depend on what happens after these models leave the laboratory.
When connected to tools, sensitive information or critical infrastructure, multi-agent AI systems can exhibit emergent capabilities and behaviors—outcomes arising from interactions among agents, tools and environments that may not be apparent in laboratory testing. AI safety must therefore cover the entire value chain from development to integration and deployment, where critical risks may otherwise go undetected and unaddressed.
When one failure becomes many
Publicly documented incidents provide only a partial picture of AI risk, particularly regarding failures that occur when AI is integrated into operations, connected to external tools or given greater authority to act. This lack of visibility matters because failures cannot remain isolated.
Shared AI models, components and services can expose multiple organizations, multiple systems and sectors to the same vulnerability. Because they cross borders, a single flaw can have international consequences. AI failures could, for example, expose sensitive information, trigger unauthorized transactions, influence irreversible healthcare decisions or disrupt entire fleets.
In 2024, a major software outage demonstrated how one flaw can disrupt millions of devices and critical services. Although not an AI failure, it illustrated the cascading risks of shared models, components and infrastructure. The scale of harm depends on what a system can do, the authority it has been granted, how widely it is deployed and how quickly failures can be detected and contained.
The question is no longer only whether an AI system can fail, but how far the consequences of such a failure could spread.
Without a clear map of the AI value chain, defined responsibilities and transparent incident reporting across the global AI ecosystem, we cannot understand the nature, frequency, severity or reach of failures or how best to prevent and contain them.
How the Forum helps leaders make sense of AI and collaborate on responsible innovation
The Centre for AI Excellence connects leaders across sectors to make sense of the rapid evolution of artificial intelligence and its implications for economies and societies. It supports informed dialogue on opportunities, risks and trade-offs linked to AI adoption.
Through trusted spaces and collaborative initiatives, the Centre enables partners to build shared understanding and work together on responsible approaches to AI innovation and use.
Policy alone is not a safety guardrail
Policies, safety commitments, codes of conduct and model evaluations matter. But none, on their own, can prevent an AI system from behaving unexpectedly or acting beyond its intended authority.
A model may perform well in testing but behave differently when connected to real-world tools, data and operational infrastructure, because testing conditions and sandboxes rarely reflect the complexity of deployment environments.
This distinction between model and system safety erodes as AI autonomy advances faster than effective controls. Safeguards should limit the system’s blast radius and support prevention, containment and recovery.
Organizations must set themselves access limits, require human approval and maintain the ability to stop AI systems. Developers should disclose capabilities and failure modes, while independent evaluations test whether safeguards remain effective under realistic conditions and system updates.
Safety cannot simply be promised, it must be demonstrated, and accountability must be built into that process. Developers, integrators and deployers need clearly defined responsibilities and records detailed enough to reconstruct failures. Organizations must be able to investigate, contain, recover from and, when necessary, suspend affected systems.
The ability to contain and recover from failures is as important as the ability to prevent them.
Three priorities for shared action
No single actor can address these challenges alone: developers understand model capabilities and limitations, integrators and deployers determine how systems are used, independent experts scrutinize safety claims and governments establish accountability frameworks.
Three priorities offer a practical foundation for cooperation and continuous lifecycle monitoring after deployment:
1. Govern what AI systems can actually do
Frontier AI evolves as models become more capable, accessible and autonomous, challenging governance based on fixed labels or training compute.
Fixed labels and training metrics do not capture the risks of deployed AI. Assessments should focus on what the complete system can do, under which conditions, with what autonomy, access and safeguards and be revisited when these change.
Governance must keep pace with what AI systems can do, not only with what they were originally designed to do.
2. Make AI incidents visible
AI incidents must be reported and shared more consistently.
Today, inconsistent reporting makes it difficult to compare failures, identify recurring vulnerabilities or tell whether an isolated incident is the first sign of a broader problem. Governments, developers, integrators and deployers should agree on common ways to report incidents such as unexpected autonomous actions, security breaches, containment failures and near misses.
Shared definitions and severity scales would let organizations learn from each other's failures across sectors and borders. Aviation did this decades ago: confidential, no-blame reporting of near misses is one reason flying became so safe.
Existing initiatives, including the OECD AI Incidents Monitor, the EU AI Act’s serious-incident reporting provisions and the Forum’s Agent Capability and Authorization Profile provide useful foundations for moving from reactive reporting to proactive authorisations, but greater global cohesion is still needed.
Effective reporting also depends on audit trails of what the system did, which tools it used, when and on whose authority. Without them, nobody can tell a model failure from an integration error, a weak safeguard or a bad deployment decision.
This does not require full public disclosure. Sensitive commercial and security information should remain protected, while ensuring the people responsible for preventing and responding to harm have the information they need.
3. Require evidence that safety measures work
Organizations should not rely solely on supplier assurances, policies or benchmarks. Safety should be demonstrated for each system and context through a structured evidence base, similar to a safety case in aviation or nuclear power showing anticipated failures, tested safeguards and response measures.
Responsibility is shared: developers document model capabilities, limitations and changes; integrators test the full system under realistic conditions; and deployers monitor operation, near misses and suspension triggers. Scrutiny should reflect potential impact radius, reversibility, autonomy and access: a document summarizer may require self-assessment, while an agent moving money or operating equipment warrants independent testing before deployment and after significant changes. Evidence must remain current as models, permissions and operating conditions evolve.
The objective is not to eliminate all risk, but to demonstrate that risks are understood, safeguards hold under realistic conditions and failures can be contained.
From reacting to preparing
Frontier AI could accelerate scientific discovery, engineer complex systems, raise productivity and help address global challenges. These benefits depend on organizations’ trust that they can deploy increasingly capable systems responsibly while maintaining control.
The international debate often centres on whether AI development should accelerate or slow. That question matters, but it should not overshadow the practical measures available today.
Better incident reporting, independent safety evidence and capability-based assessments can help identify risks earlier and respond faster. Delivering them requires cooperation among frontier laboratories, deploying organizations and governments. The goal is to ensure that our ability to understand and manage AI’s consequences keeps pace with its development.
International collaboration should turn these priorities into practical safeguards and the time to act is now.
