Can You Trust AI For Medical Advice? Here's What This Study Found

Artificial

intelligence tools have become a key part of our lives. From coding to script writing, AI has been used in many industries these days. These tools such as ChatGPT could be highly effective at answering health-related queries and assessing the data of patients to suggest symptoms.Amulya Yadav, associate professor at the College of Information Sciences and Technology at Penn State, found that large language models (LLMs) answered health-related queries correctly nearly 76 per cent of the time when evaluated by board-certified physicians. This research comes at a time when people are turning to AI chatbots for even health advice, raising questions about whether AI can be trusted for self-diagnosis.

AI “doctors” get it right 76% of the time

Why Did Researchers Conduct This Study?

According to Amulya Yadav, the team wanted to understand whether AI-powered chatbots have the potential to become the next generation of online self-diagnosis tools."For a long time now, we understand how people have been using things like Google to self-diagnose their health symptoms," Yadav said. He highlighted that the advanced AI tools have prompted researchers to find out whether LLMs for health advice could be useful or problematic. He asserted, "We wanted to understand how much more problematic, or maybe not, maybe it's more useful, would it be if people started using large language models for self-diagnosing."

How Accurate Were AI Chatbots?

As per Yadav, researchers collected over 200 health-related prompts from participants and submitted them to various LLMs including OpenAI’s ChatGPT. Yadav noted that LLMs were found to be approximately 76 per cent accurate. He mentions that ChatGPT delivered the strongest performance.

Could AI Help People Without Access To Doctors?

Notably, Yadav suggested that one of a major findings relates to healthcare accessibility. The professor pointed to global estimates suggesting that around half of the world's population lacks adequate access to healthcare services."The fact that anybody around the world can use these LLMs to get health advice is correct 76 per cent of the time. This seems to be a wonderful thing," he said. Yadav argued that AI could serve as a valuable resource for people living in regions where medical professionals are difficult to access."If the situation is that a person does not have access to any doctor, does it not make sense for them to get some imperfect help from a language model?" Yadav asked.

Should People Trust AI For Medical Advice?

Despite the promising results, researchers repeatedly warned against relying solely on AI for medical decisions. "Large language models at this point of time are not as accurate as a human physician," Yadav stressed.He urged users to exercise caution when using chatbots for self-diagnosis and stressed that healthcare professionals remain far better equipped to interpret symptoms and medical recommendations. "We should be very cautious in using large language models to self-diagnose," he said. According to Yadav, doctors are more capable of identifying when an AI response may be inaccurate or misleading.