AI red lines: the opportunities and challenges of setting limits

As AI capabilities continue to advance, ensuring systems remain safe, ethical and aligned with societal norms is a critical concern. Behavioural red lines are a proactive proposal to address unacceptable AI behaviours that pose serious risks.

Red lines demonstrate specific boundaries that AI systems must not cross, such as engaging in unauthorized self-replication, breaking into computer systems, or enabling the development of weapons of mass destruction (WMDs). A similar concept was explored in the International Dialogues on AI Safety and subsequently published in the Beijing statement.

Such red lines are not intended to be exhaustive in terms of delineating all forms of undesirable behaviour by AI systems. By establishing clear behavioural limits, red lines may serve as a critical starting point for defining unacceptable AI behaviours and a foundation for building provably safe and beneficial AI systems.

There are many opportunities and challenges associated with defining, complying with and enforcing behavioural red lines for AI. Here we highlight key examples, desirable properties of red lines, and the mechanisms needed to ensure AI systems adhere to these critical boundaries.

Red lines fall into two broad categories: unacceptable AI uses and unacceptable AI behaviours.

Unacceptable AI uses are linked to constraints on how humans might misuse AI technologies. The EU AI Act, for example, imposes restrictions on how humans may use AI-based video surveillance tools. Unacceptable AI behaviours are actions that AI systems must not take, regardless of whether or not the action is in the service of a human request. For example, an agentic AI system must not engage in improper surveillance via webcams, even if doing so would help to satisfy a legitimate human request for help.

The governance mechanisms for these two categories are somewhat distinct. Governance for usage red lines, like other constraints on human behaviour, would typically be ex post, imposing penalties for violations. Governance for behavioural red lines, like other constraints on technological systems, might involve a combination of ex ante (e.g. design requirements) and ex post, depending on the severity of harm and feasibility of prevention.

Behavioural red lines are particularly important for addressing unintended harms arising from AI systems that can act and influence the real world with greater degrees of autonomy. By identifying and prohibiting these unacceptable behaviours, stakeholders can build a foundation for a safer AI ecosystem while encouraging the development of tools to monitor and enforce compliance.

The most effective and enforceable red lines would ideally exhibit three desirable properties:

For a red line to have the desired effect of advancing AI safety engineering standards, it would also need to involve non-trivial compliance challenges – well beyond simple output filters, for example. This means requiring more comprehensive safeguards, such as system-wide monitoring, rigorous testing and enforceable accountability measures to ensure AI behaves as intended in high-stakes situations.

We explored a wide range of possible behavioural red lines. All of those listed below involve clearly undesirable behaviour, but, as we discuss, some of them may not satisfy all of the desirable properties listed above.

We stress that the inclusion of a red line on this list does not imply that we advocate for its implementation in regulations. Nor does it imply that AI systems have already crossed it. We provide hyperlinks to examples of violations that have already occurred or have been shown to be feasible in tests.

As noted above, not all of these red lines conform entirely to the three criteria listed. For example, “advising on weapons of mass destruction” is difficult to define clearly, as what counts as effective advice depends on the intent and background knowledge of the user. Similarly, there are several grey areas in the law of defamation – which, moreover, is not universally defined in the same way across jurisdictions. What counts as discrimination is also not universally agreed, in that protected categories vary widely by jurisdiction and application context.

There is much work to be done to reach agreement on which red lines are most suitable and on exactly how they should be defined and implemented in regulations. Concerns are also linked to the technological feasibility of compliance and to the adequacy of current mechanisms for enforcement.

Ensuring that AI systems adhere to behavioural red lines requires a comprehensive approach combining both compliance mechanisms and enforcement tools.

In terms of compliance, ex-ante regulation refers to measures that are applied before an AI system's deployment, such as registration, licensing and certification. Certification requirements might include a safety case: as the UK Ministry of Defence defines it, “a structured argument, supported by a body of evidence, that provides a compelling, comprehensible and valid case that a system is safe for a given application in a given environment”.

The gold standard for ensuring properties of software systems is formal proof, but other approaches are possible. In addition to designing systems that refrain from crossing red lines, it is a good idea to add built-in safeguards to prevent actual breaches of red lines in cases where the safety case breaks down. This combined preventative approach mirrors established safety practices in high-risk industries such as aviation and nuclear energy.

Complementing preventative measures is ex-post regulation, which involves imposing consequences after an AI system breaches established red lines. Consequences could include fines, liability or other penalties aimed at deterring future violations. Organizational oversight is another critical pillar that may involve ethics boards, collaborative governance initiatives and transparency reporting. For high-stakes AI applications, however, ex-post regulation alone might not be sufficient and should be supplemented by proactive measures to ensure safety and prevent undesirable outcomes.

Another crucial mechanism is continuous monitoring, which involves real-time tools to detect and flag violations, supported by both automated audits and human oversight. This monitoring exists within a context of shared accountability, where developers, deployers and end users all bear responsibility for ensuring compliance and fostering a collaborative approach to safety.

In addition to compliance requirements, enforcement mechanisms play a crucial role. Technical enforcement measures include fail-safe mechanisms such as automated shut-down protocols that can be invoked when monitoring systems detect a violation.

Actual enforcement, however, faces several challenges. These include jurisdictional variability, resource limitations and the risk of overly punitive measures that could limit innovation in some cases. The rapid development of frontier AI systems further complicates this landscape, requiring frameworks that are flexible enough to adapt to new and emerging risks, while maintaining effective oversight and control.

The current approach to AI safety often involves retroactive attempts to reduce harmful tendencies after a system has been developed. This reactive model may be insufficient for addressing the risks posed by advanced AI, especially when such systems are displaying greater degrees of autonomy.

Behavioural red lines for AI could contribute to encouraging a proactive shift toward making safe AI by design. By requiring developers to provide high-confidence guarantees of compliance, similar to those expected in high-risk industries such as nuclear energy and aviation, establishing behavioural red lines for AI could contribute to more advanced safety engineering, greater predictability and verifiability, and improved regulatory collaboration across jurisdictions. This in turn will foster trust and ensure that AI systems serve as tools for progress, not as sources of harm.

The following members of the Global Future Council on the Future of AI contributed to this piece: Stuart Russell, University of California, Berkeley; Edson Prestes, Federal University of Rio Grande do Sul, Brazil; Mohan Kankanhalli, National University of Singapore; Jibu Elias, Mozilla Foundation; Constanza Gómez Mont, C Minds; Vilas Dhar, Centre for Trustworthy Technology, World Economic Forum; Adrian Weller, University of Cambridge and The Alan Turing Institute; Pascale Fung, Hong Kong University of Science and Technology; Karim Beguir, Co-founder and CEO of InstaDeep.