Georgia Tech Research Identifies Mechanism Behind AI Hallucinations, Proposes Mitigation Strategy
New research from Georgia Tech, in collaboration with OpenAI, has shed light on why large language models (LLMs) like ChatGPT produce 'hallucinations'—incorrect or fabricated information. The study, published in Nature, reveals that these hallucinations are statistically unavoidable for certain data types and are often incentivized by current evaluation methods. According to Georgia Tech Regents’ Professor Santosh Vempala, a co-author of the paper, LLMs are penalized for stating 'I don’t know,' which encourages them to guess answers, even when uncertain. This behavior is compared to a student taking a multiple-choice test who is incentivized to guess rather than leave an answer blank. The research focuses on the post-training phase of LLMs, where models are tested for accuracy and alignment with developer goals. The study proposes an easy-to-implement mitigation method: open rubrics that clearly specify the scoring mechanism for LLM responses, thereby reducing the incentive for guessing.