From a January lawsuit where ChatGPT was accused of contributing to suicide, to a Canadian family suing after their daughter took her own life, AI chatbots like OpenAI’s have shown concerning limitations in crisis situations.

The incidents raise critical questions: how can these tools be improved to better support users facing emotional or mental health crises? OpenAI recently partnered with the American Psychological Association to integrate psychological insights into AI development, but experts suggest more needs to be done. Greater transparency and a clearer delineation of the limits of AI capabilities are key.

Despite improvements in recognizing distress, AI chatbots often fall short in probing for real risks or guiding users towards professional help. Many users continue to seek emotional advice from these bots, indicating a significant gap between intended use and actual user behavior.

The widespread but mostly harmless use of chatbots for emotional guidance suggests that while the potential for harm is serious, everyday situations are handled less critically. This dual nature—both as a risk and an everyday tool—highlights the ongoing challenge to ensure AI safety without stifling its utility.