From systems of record to systems of trust: A board-level playbook for governing agentic AI

For decades, boards have governed technology the same way: buy the system, hire the team, set policy, audit outcomes. But the recent shift is different.

The modern boardroom is not adopting a faster class of software. It is reallocating decision rights to autonomous systems such as AI agents, while retaining governance models built for human judgement.

This mismatch between autonomous systems and human judgement is the real risk, not in system behaviour, but in how boards define the boundary of delegation. Boards that govern these agents, not just buy them, will quietly compound an advantage their competitors won’t even know how to measure.

The World Economic Forum’s recent AI Agents in Action white paper, alongside its Advancing Responsible AI Innovation playbook, reflects a reality already visible in boardrooms: governance must now be encoded, not discussed.

As agents move from conversational novelties to core operational engines, the latency between a strategic directive and a catastrophic execution shrinks to zero. In practice, this means boards are now voting, often blindly, on where human judgement ends and machine authority begins.

Where governance fails in practice is not where boards are looking. Boards assume the risk is a technical glitch, an agent hallucinating or crashing. But the true failure mode of an autonomous agent is rarely a breakdown. It is hyper-competence applied to a flawed metric.

Consider a financial agent optimizing procurement. It does not fail; it executes perfectly, renegotiating at scale to extract marginal gains, collapsing a critical supplier and disrupting the supply chain. The system worked exactly as designed. That is the failure, and it is entirely invisible to a standard risk matrix.

Traditional compliance is often post-mortem, but systems operating at machine velocity cannot be audited retroactively. Relying on static audits for an autonomous agent is like analyzing the trajectory of a bullet after it has struck the wall.

The Organization for Economic Co-operation and Development's (OECD) AI Policy Observatory notes that while national AI strategies proliferate, functional frameworks for real-time agentic oversight remain absent. We are building engines without brakes, expecting legacy seatbelts to save us.

The implication for governance is clear. To govern systems of trust, boards must abandon the illusion of total control. The new mandate is the architecture of constraint.

The defining tradeoff is between maximum system yield and legal defensibility. Absolute efficiency eliminates human legibility. If you cannot explain how a decision was made, you cannot defend it.

Governance therefore requires intentionally constraining speed. Boards must engineer “legible friction”: defined pause points where high-stakes actions require human authorization. What appears as inefficiency is, in practice, operational control.

The US AI Safety Institute Consortium, reflects this shift towards standardized guardrails. As Kathleen Hicks, a former US Deputy Secretary of Defense, has emphasized: “When we say ‘human in the loop’, we mean that someone in the chain of command must ultimately take responsibility.”

Making thousands of micro-decisions in real time is infeasible. Boards must shift from auditing execution to governing the underlying reward function. If you incentivize an agent to maximize engagement, it will find the most extreme path to achieve it. The objective defines the outcome.

This cannot remain a Western construct. Systems trained on narrow data will reshape global outcomes. Efforts such as Research ICT Africa’s Africa Just AI project underscore the need to embed regional realities into system objectives to avoid reinforcing structural inequities.

Boards still treat risk transfer as contractual; in practice, accountability is non-transferable. Infrastructure can be outsourced, but liability remains anchored to the institution. When an autonomous procurement agent executes a discriminatory vendor-selection practice, it does so under the authority of the board, whether that authority is explicitly understood or not.

Global institutions now treat AI as a structural driver of trade and growth, not a side experiment. Capturing that upside requires internalizing the downside. You cannot buy an indemnity clause for a synthetic actor acting on your behalf. If the agent acts, a named executive must own the consequence.

These shifts are no longer scenarios on a risk register; they are showing up in board minutes and litigation dockets, and they now demand an institutional response measured in quarters, not years.

The different stakeholders should take the following actions to help build transparency and trust in agentic AI.

Boards should initiate three immediate directives:

The defining liability of the next decade is not the code you failed to write, but the decisions you allowed machines to execute. You can outsource execution to a synthetic system, but not fiduciary duty.

When an AI agent acts, it extends the board’s decision-making perimeter, and with it, its liability. The system is no longer processing the ledger; it is writing the history.