How healthcare can build a human-in-command approach to AI

Every sector of the economy is being transformed by artificial intelligence but now, more than ever, the need for safeguards has come into sharp focus, as has the need to do this without curbing AI's immense potential, such as in the area of drug discovery.

The healthcare sector has an opportunity here – and a duty – to lead the way on deploying AI safely.

Healthcare organisations now face a practical challenge: qualifying what an AI system can safely do, determining where human authority should sit and adapting those controls as the technology evolves.

Before determining how much authority an AI system should have, organizations need to understand the conditions under which it can be trusted for a particular task. Healthcare already uses systems of qualification, evidence, defined scopes of use, monitoring and accountability. AI requires mechanisms suited to technology whose capabilities and behaviour can change quickly.

Tools such as data cards, model cards and risk analyses can document what shaped a system, how it was evaluated, its intended context of use, known limitations and failure modes and when additional verification is required. The US Food and Drug Administration defines model cards as a transparency mechanism, while the Coalition for Health AI has developed an applied model card for healthcare.

Qualification should examine populations evaluated, conditions where performance deteriorates, evidence relied upon, behaviour with incomplete or conflicting information and the effect of changes to the model, evidence or workflow. US health IT policy is already moving toward greater algorithm transparency in certified health technology.

Qualification is one layer of safety. The next is understanding the decision process the model operates within. An AI-enabled healthcare workflow can move from underlying data through context and evidence, reasoning, recommendation, action and outcome.

Risk can enter at any stage; data may be incomplete and relevant evidence may not be retrieved. Furthermore, a system may misunderstand patient context, apply guidance outside its intended circumstances or produce reasoning inconsistent with the evidence.

Subject-matter experts are essential to identifying plausible failures, consequential exceptions and the points where professional judgment can materially change a decision. Good machine learning practice similarly emphasises multidisciplinary expertise and attention across the total product lifecycle.

Once potential failures are understood, three dimensions can help calibrate oversight: the likelihood of error, the consequence if it occurs and whether the resulting action is reversible.

A low-risk, reversible administrative task may support substantial automation but a consequential clinical recommendation based on ambiguous information requires stronger verification.

Different risks also require different checkpoints:

The system should also evaluate human involvement. A human checkpoint does not automatically improve an AI-assisted decision. The sequence of human and AI reasoning, the information presented to the reviewer, the reviewer’s expertise and the design of the interaction can all affect the result.

Therefore, organizations should test whether the combined human-AI workflow improves decision quality and safety under the conditions in which it is deployed.

We propose five elements for an operating model which places the human in command of a human-AI workflow:

AI capabilities change rapidly. As clinical evidence evolves, patient populations shift, policies change and workflows are redesigned. Qualification and oversight therefore need to be continuous. For example, research in Nature Medicine has argued for recurring local validation of health AI because performance can vary across settings and over time.

Organizations should therefore monitor error patterns, overrides, escalation rates, disagreements, model and data drift and downstream outcomes. Persistent clinician overrides may reveal a limitation missed during initial evaluation.

Repeated escalation of one type of case may point to a need for additional evidence, training or workflow redesign. In that respect, real-world deployment should continually update where a system performs reliably, where its boundaries remain and how much authority it should have.

For healthcare leaders, three key principles stand out when navigating a human-AI workflow to get the best out of AI while maintaining safety:

Healthcare can contribute a practical model to the wider AI governance debate. The objective is to build systems in which capabilities are understood, risks are anticipated, consequential decisions remain interruptible and human expertise is positioned where it creates the greatest value.

Human-in-command means qualifying the technology, mapping its risks, designing the right checkpoints, preserving meaningful human authority and continuously learning as both technology and evidence evolve.