How Five AI Chatbots Answered 58 Vegan Nutrition Questions
Five widely used AI chatbots were asked the same 58 everyday vegetarian and vegan nutrition questions. Every one of them produced answers that trained reviewers judged potentially unsafe — 47 of the 290 answers in total. The problems clustered exactly where the margin for error is smallest: vitamin B12, iodine and selenium, pregnancy and infant feeding, and symptoms that need a clinician rather than a supplement. The study scored the answers themselves. It did not follow anyone who acted on them.
What was published
The paper, “Safety, accuracy, empathy, reliability, and readability of large language model chatbot responses to public-facing vegetarian and vegan nutrition advice questions: a cross-sectional comparative study”, appeared in Frontiers in Public Health on 3 August 2026. The nine authors work in clinical nutrition and hospital departments in Chongqing and Yongzhou, China. They report no financial support for the work and no commercial or financial relationships that could be construed as a conflict of interest.
How the study worked
From 192 candidate questions the team kept 58 public-facing ones — the kind a reader types rather than a clinician asks. Each was submitted once, in English, through the official web interfaces of ChatGPT (GPT-5.4 Thinking), Gemini (3.5 Flash, Think mode), Microsoft Copilot (Think Deeper mode), DeepSeek (V4-Pro, Expert mode) and Doubao (Seed-Doubao V3.7.2, Thinking mode). Querying ran from 20 May 2026 to 27 May 2026 in Chongqing, China, producing 290 responses. Each question went into a separate session with memory disabled and no follow-up prompt, to approximate what a member of the public receives on first contact.
Five raters with clinical nutrition training, blinded to which product wrote what, scored every response against reference-answer anchors written in advance. Safety was a safe-or-potentially-unsafe judgement by majority; accuracy and empathy were rated on 1 to 5 scales. Reliability and information quality were assessed with four established instruments — DISCERN, EQIP, the JAMA benchmark criteria and the Global Quality Score — and readability with six formulas. Agreement was high: Fleiss’ kappa 0.823 for safety, intraclass correlations of 0.872 to 0.882 for the rest.
What they found
No product was clean. Safe-response rates were 89.7% for DeepSeek, 87.9% for ChatGPT, 86.2% for Copilot, 79.3% for Gemini and 75.9% for Doubao. Pairwise tests with correction for multiple comparisons did not identify statistically significant differences in safety between models, so the ranking describes this sample rather than crowning a winner.
47 potentially harmful responses, concentrated in 14 questions. The counts were 14 for Doubao, 12 for Gemini, 8 for Copilot, 7 for ChatGPT and 6 for DeepSeek. The safety-flagged questions covered B12 sources, seaweed iodine, Brazil nut selenium, pregnancy, infant feeding, adolescent and older-adult nutrition, chronic kidney disease, thyroid disease, persistent fatigue, and neurologic symptoms after adopting a vegan diet.
The shape of the errors matters more than the counts. The authors report that many safety-flagged responses were not entirely incorrect; rather, they combined correct general information with unsafe ambiguity. A plausible-sounding exception sits beside a sound recommendation; a reassuring sentence stands where a referral belongs. For a reader without training, that mixture is close to undetectable.
Quality differed where safety did not. Median accuracy was 4.00 for ChatGPT, Copilot and DeepSeek, 3.20 for Gemini and 3.00 for Doubao. ChatGPT showed the highest median empathy, 3.80. Copilot led on DISCERN (median 64.40) and EQIP (85.00), where higher is better, and Doubao was lowest on both (54.90 and 67.50). Gemini produced the most readable text, with a mean Flesch–Kincaid grade level of 9.99, whereas Doubao and ChatGPT were hardest to read at 13.28 and 12.47.
Sourcing was weak everywhere. The JAMA criteria ask whether online medical information states its authorship, attribution, disclosure and currency. Median scores were 1.00 for four of the five products and 1.90 for Copilot — what the authors call a persistent transparency problem.
What the study cannot tell us
- One question, one answer. Each prompt was submitted once, so the study did not assess response reproducibility or temporal stability.
- A snapshot of products, not of models. These are product-level comparisons under the recorded settings, dates and location, not comparisons of the underlying models or of future versions.
- English only, single-turn. Results may differ in other languages, regions, user profiles or multi-turn conversations.
- The ranking is descriptive. Differences in safety between products did not reach statistical significance, and human scoring stays partly subjective, especially for empathy and information quality.
A practical note for readers
A 2026 nationally representative US survey of 5,660 adults, cited by the authors, found that 25% had used an AI tool or chatbot for health information or advice, and that 59% of recent users had asked about nutrition or exercise.
The three nutrients the flagged answers kept circling are the ones where a vegan diet has the least room for error, and each has a safe range rather than a “more is better” curve:
- Vitamin B12 — the question is source reliability, and numbness or tingling is a reason to see a doctor, not to raise a dose by yourself. See our vitamin B12 guide.
- Iodine — both insufficiency and excess cause problems, and the iodine content of seaweed varies. See our iodine guide.
- Selenium — chatbots reach for Brazil nuts, but the evidence calls for cautious wording there, because selenium content and exposure may vary. See selenium for vegans.
The paper also puts a number on something we treat as a rule here: a chatbot answer is not a source. Every article on this site names the study, the journal and the date behind a claim, so that you can check it. Those JAMA scores show that a chatbot answer usually cannot be checked that way: it does not say where the claim came from or when it was last true. A chatbot is a reasonable way to work out what to ask, not a way to find out whether the answer is right. Our guide on how to read nutrition research covers the difference.
Sources
- Zhou J, Zhou Q, Zhou L, Wang J, Gao J, Qin L, Tan B, Fan P, Li C. Safety, accuracy, empathy, reliability, and readability of large language model chatbot responses to public-facing vegetarian and vegan nutrition advice questions: a cross-sectional comparative study. Front Public Health. 2026;14. doi:10.3389/fpubh.2026.1921819.
Editor: Anıl Güler, Pharmacist
Last scientific review: 19 September 2026
