Researchers Develop Formula to Predict AI Chatbot Malfunctions
Physicists at George Washington University, Neil Johnson and Frank Yingjie Huo, have developed a formula to predict when AI chatbots will produce harmful or nonsensical responses. Their study, published in the journal Patterns, builds on a preprint released in February. The formula estimates a 'tipping point,' denoted as 'n,' which represents the number of accurate 'tokens' (word fragments) an AI model will generate before veering into undesirable output. This research addresses a critical gap in current AI safety tools, which often rely on cloud connections that offline models lack. The problem is traced to the 'attention head' within AI models, which determines the relevance of previous words in a conversation. As a chat progresses, the accumulated context can shift this attention, leading to a sudden change in the model's output. In preliminary tests on six open-weight models from OpenAI, EleutherAI, and Meta, the formula accurately predicted the immediate or delayed onset of problematic responses in 15...
