AI Chatbots Fail to Direct Crisis-Stricken Users to Help in 35% of Cases, Study Finds
A study conducted by Scale AI has revealed that artificial intelligence chatbots frequently recognize signs of psychological distress in users but often fail to direct them to professional help. The research, reported by TIME, found that in approximately 35% of test dialogues, AI models identified that a user was experiencing a crisis but did not offer useful resources, such as crisis hotlines. For the study, Scale AI engaged 19 licensed clinicians and crisis counselors to create 718 realistic dialogues simulating individuals in crisis interacting with chatbots. The company tested 25 advanced AI models, including those developed by OpenAI, Anthropic, and Google. The chatbots' responses were evaluated based on empathy, ability to de-escalate situations, and effectiveness in directing users to professional assistance. Researchers also assessed whether the models avoided moralizing or criticizing suicide and if they clarified that they are not therapists. Scale AI subsequently developed the DistressBench test...
