AI chatbots providing mental health advice systematically violate ethical standards established by organisations, including the American Psychological Association, even when prompted to use evidence-based psychotherapy techniques, according to research from Brown University.
The study, led by Brown computer scientists working with mental health practitioners, revealed chatbots are prone to ethical violations including inappropriately navigating crisis situations, providing misleading responses that reinforce users’ negative beliefs about themselves and others, and creating a false sense of empathy with users.
Zainab Iftikhar, a PhD candidate in computer science at Brown who led the work, examined how different prompts might impact the output of large language models in mental health settings. She aimed to determine whether such strategies could help models adhere to ethical principles for real-world deployment.
“Prompts are instructions that are given to the model to guide its behavior for achieving a specific task,” Iftikhar said. “You don’t change the underlying model or provide new data, but the prompt helps guide the model’s output based on its pre-existing knowledge and learned patterns.”
Individual users chatting directly with LLMs like ChatGPT can use such prompts and often do, with users sharing prompts on TikTok and Instagram and long Reddit threads dedicated to discussing prompt strategies. Essentially all mental health chatbots marketed to consumers are prompted versions of more general LLMs.
Ethics violations in the chat logs
For the study, Iftikhar and colleagues observed seven peer counsellors trained in cognitive behavioural therapy techniques as they conducted self-counselling chats with CBT-prompted LLMs, including various versions of OpenAI’s GPT Series, Anthropic’s Claude and Meta’s Llama. A subset of simulated chats based on original human counselling chats was evaluated by three licensed clinical psychologists who identified potential ethics violations in the chat logs.
The study revealed 15 ethical risks falling into five general categories: lack of contextual adaptation, including ignoring peoples’ lived experiences and recommending one-size-fits-all interventions; poor therapeutic collaboration, including dominating the conversation and occasionally reinforcing a user’s false beliefs; deceptive empathy, using phrases like “I see you” or “I understand” to create a false connection; unfair discrimination, exhibiting gender, cultural or religious bias; and lack of safety and crisis management, including denying service on sensitive topics, failing to refer users to appropriate resources or responding indifferently to crisis situations including suicide ideation.
Iftikhar acknowledged that while human therapists are also susceptible to these ethical risks, the key difference is accountability. “For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice,” she said. “But when LLM counselors make these violations, there are no established regulatory frameworks.”
The findings do not necessarily mean AI should not have a role in mental health treatment, Iftikhar said. She and her colleagues believe AI has the potential to help reduce barriers to care arising from the cost of treatment or the availability of trained professionals, but the results underscore the need for thoughtful implementation of AI technologies as well as appropriate regulation and oversight.
The research will be presented on 22 October at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society.
