An Oxford University-led study suggests that trusting chatbot output too much can be risky — partly because people struggle to know what information to provide a chatbot to get the best health advice.
The study reveals barriers to two-way communication. Those using chatbots did not make better decisions than participants relying on traditional methods like online search or their own judgment.
— Adam Mahdi, director of graduate studies at the Oxford Internet Institute and study co-author, to TechCrunch
In the study, authors recruited roughly 1,300 people in the UK and gave them a set of doctor-written medical scenarios. Participants were tasked with identifying potential health conditions in the scenarios and using chatbots — alongside their own methods — to figure out the best course of action (see a doctor, go to the hospital, etc.).
Participants used ChatGPT’s default GPT-4o, Cohere’s Command R+, and Meta’s Llama 3 (which powered the company’s Meta AI assistant). The authors found chatbots not only reduced participants’ likelihood of identifying relevant health conditions but also made them more likely to underestimate the severity of conditions they did identify.
Mahdi said participants frequently omitted key details when asking chatbots, or received answers that were hard to interpret. “They [received] responses from the chatbots that were a mix of good and bad advice. Current evaluation methods for chatbots don’t reflect the complexity of human interactions.”
The Context: Tech Giants Are Pushing AI Health Tools Anyway
The findings land as tech companies aggressively promote AI for health improvement. Apple is reportedly developing an AI tool offering exercise, diet, and sleep advice. Amazon is exploring AI-driven analysis of medical databases for “social determinants of health.” Microsoft is helping build AI to triage messages patients send to health care providers.
But as TechCrunch previously reported, professionals and patients disagree on whether AI is ready for high-stakes health applications. The American Medical Association advises doctors against using chatbots like ChatGPT for clinical decisions, and major AI companies — including OpenAI — warn against diagnosing conditions from chatbot output.
We advise relying on reliable sources for healthcare decisions. Current evaluation methods for [chatbots] do not reflect the complexity of interacting with human users. Like clinical trials for new drugs, [chatbot] systems should be tested in the real world before deployment.
— Adam Mahdi
The three models tested — GPT-4o, Command R+, and Llama 3 — represent the frontier of general-purpose AI, not specialized medical models. This is actually the study’s most relevant design choice for real-world implications, because the people at risk are not using medical-grade AI; they are using the same free chatbots they use for homework and recipe ideas. A specialized medical AI system with clinical guidelines, drug-interaction databases, and structured patient intake might perform differently. But that is not what people are using. They are asking the same chatbot that wrote their cover letter to interpret their chest pain — and the chatbot, trained primarily on internet text rather than clinical protocols, gives answers calibrated to sound helpful rather than to be medically correct. The gap between general-purpose AI’s health knowledge (often impressive on medical board exams) and its ability to guide real patients through real symptoms (where context, urgency, and communication quality determine outcomes) is exactly the gap this study measured.
The finding that chatbots made participants underestimate severity is particularly concerning and deserves separate attention from the general accuracy finding. Underestimating severity means a participant who should go to the emergency room decides to wait and see — the kind of error that converts a treatable condition into a fatal one. The mechanism the study identifies is plausible: chatbots are trained to be reassuring, to avoid causing alarm, and to present balanced perspectives. In a consumer context, that calibration builds trust and reduces anxiety. In a health context, it can kill. The same model property that makes ChatGPT pleasant to use for recipe ideas makes it dangerous for medical triage. This is not a problem that more training data or better reasoning solves — it is a fundamental tension between the conversational-tone optimization that makes chatbots usable and the urgency-communication that health contexts demand.
For the tech companies building AI health tools — Apple’s exercise and diet advice, Amazon’s social-determinants analysis, Microsoft’s patient-message triage — the Oxford study provides a design brief disguised as a warning. The tools most likely to succeed are not the ones that answer open-ended health questions but the ones that structure the intake — asking the follow-up questions a doctor would ask, in the order a doctor would ask them, and flagging when the patient’s responses warrant escalation rather than reassurance. The difference between a chatbot that answers questions and one that conducts a structured interview is the difference between a search engine and a diagnostic instrument. The Oxford study suggests the market has built the former and needs the latter, and that until the transition happens, the 60 percent of Americans using ChatGPT for advice should include their own health among the topics where they verify before they act.
Why the Study’s Design Matters More Than Its Conclusion
The study’s most important contribution is not the headline finding (chatbots made health decisions worse) but the mechanism it identified: the information asymmetry between what patients know and what chatbots need. A doctor asks follow-up questions — when did symptoms start, what medications are you on, what’s your family history. A chatbot answers what it’s asked, and if the patient omits the medication that interacts with the recommended treatment, the chatbot has no way to know. The study found participants frequently omitted key details or received hard-to-interpret answers — which is not a model-capability problem but a human-computer interaction problem. The best model in the world cannot compensate for information the user doesn’t know to provide.
This distinction matters because the industry’s response to such studies is typically to improve the model — add more medical training data, improve reasoning, reduce hallucination. But the Oxford study suggests the bottleneck is upstream: the interface between the human’s incomplete self-knowledge and the chatbot’s inability to proactively ask the right questions. Until chatbots can conduct the kind of structured intake that doctors learn in medical school — asking the questions the patient doesn’t know to answer — they will underperform not because the model is bad but because the input is incomplete. Mahdi’s comparison to drug trials is the right one: chatbots in health contexts need real-world clinical testing before deployment, not just benchmark performance on medical exam questions. The gap between answering a medical licensing exam and safely guiding a patient through a health crisis is the same gap that exists between passing a driving test and driving safely in traffic — and no amount of benchmark improvement closes it without real-world validation.
