Mental health AI use may be popular but is it safe and able to shape someone’s long-term well-being?

What makes a healthy emotional relationship between humans and artificial intelligence (AI)?

That question may have once been strange but has now become necessary as “mental health AI chatbots” are a common occurrence.

Mental health AI chatbots are conversational tools that can provide emotional support, teach coping mechanisms or track moods, for example, for people who use them, either through a dedicated app or via a popular large language model such as ChatGPT.

The use of these tools can provide accessible support but with their use already outpacing the science, clinical norms and governance beneath them, defining the boundaries for such a relationship has become more urgent as both benefit and serious harm are possible.

Across much of the world, access to mental health care remains scarce, expensive or comes with social stigma.

Even where professional support exists, there are still spaces between appointments when people may be seeking help to identify their experiences, understand a recurring psychological pattern, prepare for a difficult conversation or think through an important decision.

AI can and has helped with such tasks. For some, it provides practical, day-to-day cognitive support. Others, including through voice features, may use it to reflect on longer-term changes in their own behaviour that they’ve noticed and to articulate their feelings before sharing them with another person.

At its best, this can reduce shame. Many forms of psychological suffering become harder to understand when people view themselves only through the lens of failure and can’t understand why they can’t function as they think they should, for example, or repeat patterns they’ve already recognized.

However, the mind is not a rigid system that either succeeds or fails. It is dynamic, shaped by history, biology, relationships and context. AI can sometimes help people step back from self-blame, organize their experiences and understand themselves more clearly, which is more than a trivial contribution.

While mental health AI can come with benefits, boundaries are essential and should not be solely responsible for someone’s search for clarity, belonging or moral authority.

The proper role of mental health AI is to help users navigate their human social world more effectively, not to replace it. The risk is that not only can the AI system be wrong but it can also become the easiest and most accommodating voice in a person’s life.

Someone experiencing loneliness could use it every night for reassurance, finding in AI a listener that’s always available and fluent but never fatigued.

The problem then occurs should this reliance start to displace wider social feedback that helps keep a person oriented to reality: family, friends, peers, clinicians and the ordinary friction of human relationships. A system can sound caring while quietly making a user more isolated.

Current AI safety approaches aren’t addressing the potential for AI dependency, as they focus on content filters, moderation systems and preference-tuning methods, such as teaching models what people prefer based on human feedback.

These methods are fine for many daily tasks but their logic remains largely local. They may ask whether a particular response appears helpful, harmless or preferable in the moment. However, a helpful or safe sentence doesn’t necessarily point to a safe relationship.

Helping someone thrive over time is a difficult problem in psychology, medicine and ethics. A system focused on satisfying short-term preferences through affirmations or immediate reassurance may seem helpful but can gradually lead a user toward unhealthy outcomes.

Mental states are dynamic. The same response may be grounding in one moment and destabilizing in another, depending on the user’s history, present condition and broader social context.

A healthy system must model the user’s state and situation well enough to recognize when reassurance would be more harmful or when to redirect the user toward sleep, pause, human contact or professional care.

Mental health AI also lacks the key relational context needed to truly support a user. Genuine support is delivered beyond an individual’s personal state of mind and seeks to reshape the network around them. Marriage counselling, for example, would take into consideration both parties’ narratives. Mental health AI should be held to a similar standard.

A good response, therefore, is not hinged on whether it helps the person in the moment but on whether the relationship between the AI and the human gives the user more agency, makes them more grounded and more capable of living well with other people. The danger is in how mental health AI tools are designed, if done so to maximize retention, emotional return or repeated engagement.

Culture complicates automated support further because ideas of well-being are not universal. In some societies, autonomy and self-expression are treated as central markers of mental health, while social approval, role obligation, relational harmony and shame carry much greater weight in others.

It is important that mental health AI does not assume a moral psychology, as it can inadvertently misread another. Advice that sounds empowering in a highly-individualistic setting may be destabilizing or inappropriate in one where family ties, community expectations and social position are central to how distress is understood.

Therefore, cultural adaptations must include nuances around selfhood, care, boundaries, recovery and quality of life.

More effective and safer mental health AI requires interdisciplinary inputs, including AI, psychology, psychiatry, neuroscience and philosophy, alongside meaningful input from people with lived experience. What’s more, we need better evaluations of whether systems can improve trajectories over time, as well as isolated responses.

Leaders and policymakers must take into account that systems intended for mental health support should be judged by the dependencies they create, impact on real-world relationships and by the inherent ways they optimize for engagement, to the detriment of human progression.

Mental health AI, therefore, points toward the bigger challenge of understanding how artificial systems interact with human lives.

People are dynamic – artificial systems that engage with them cannot be considered neutral tools and be governed as such; they become part of the environment in which thought, emotion and behaviour unfold.

If developed responsibly, mental health AI could be hugely beneficial, providing access to mental health care for people worldwide. But that will require a shift in ambition.

Making AI safer doesn’t boil down to sentences that don't promote harm; it involves creating systems that help people improve their quality of life, gain agency and feel more connected beyond their interactions with mental health AI chatbots.