What's Happening?
A recent study, based on 300 product-fact queries about perfume notes, found that AI answer engines like ChatGPT, Perplexity, and Gemini exhibit inconsistent accuracy, even when they agree on an answer. The study, conducted by Scento, revealed that while
median recall (returning official notes) was high for all three engines (1.00), agreement on the full, exact answer was rare. For instance, out of 99 perfumes, all three engines returned every official note for only 59, and matched the official list exactly for just 39. In about 20% of cases, all three engines returned every official note but also added incorrect information. The study highlighted that agreement among AI engines does not guarantee accuracy, especially when systems draw from the same pool of web text, as a majority vote may reflect widespread misinformation rather than current, verified facts.
Why It's Important?
This finding has significant implications for U.S. consumers, businesses, and developers relying on AI for product information. For consumers, it means that AI-generated product descriptions, even if consistent across multiple platforms, may contain inaccuracies, leading to misinformed purchasing decisions. For businesses, particularly those in e-commerce and product marketing, relying solely on AI for content generation without robust fact-checking can damage brand credibility and lead to customer dissatisfaction. Developers of AI agents and shopping assistants must recognize that consensus among models is not a reliable indicator of truth. This necessitates a shift towards more rigorous evaluation metrics beyond simple recall, incorporating precision, exact match, and structural accuracy, and emphasizing grounding facts in owned, verifiable sources.
What's Next?
The study suggests several critical next steps for developers and businesses. They should prioritize using consensus to identify disagreement rather than confirm truth, treating outliers as alarms. Implementing strategies to ground owned facts in owned sources, such as using schema.org Product markup, is crucial for ensuring accuracy. Evaluation metrics should expand beyond recall to include precision, exact match, and structural correctness. Furthermore, deterministic grading with rule-based systems, where reference data exists, is recommended for auditable and repeatable checks. The study also advises running queries multiple times to account for run-to-run variations in AI answers. These measures are essential to improve the reliability of AI-generated product information and build trust with users.
Beyond the Headlines
The study's findings delve into the deeper implications of AI's role in information dissemination, particularly concerning factual accuracy. It highlights a fundamental challenge: AI models, even advanced ones, can perpetuate and amplify widely repeated but incorrect information if their training data is flawed or outdated. This raises ethical questions about the responsibility of AI developers and platforms to ensure the veracity of the information they provide, especially for critical product details. The observation that accuracy varies by product segment (e.g., designer vs. niche perfumes) suggests that a 'one-size-fits-all' approach to AI evaluation is insufficient. This calls for more nuanced testing methodologies that consider the specific domain and the nature of the facts being queried, pushing the industry towards more sophisticated and context-aware AI validation processes.













