13 Assessing the Quality and Reliability of Online Health Information for Breast Pathology: Search Engines Versus Large Language Models

Background

With mainstream adoption of large language models (LLMs), more patients are using LLMs for health information. With millions of daily users, LLMs are becoming a primary information source and may eventually replace traditional search engines. Given the growth of LLM use, we aimed to compare the quality and reliability of information from conventional search engines, ChatGPT, and OpenEvidence (OE), a restricted-access medical LLM, across breast pathologies to improve patient education and determine whether patients need access to specialized medical LLMs.

Materials and Methods

Google was the conventional search engine used; searches were performed for 13 benign, high-risk, premalignant, and malignant breast pathologies. Using the DISCERN tool, a validated measure of health information quality and reliability, 2 independent reviewers evaluated websites from the first 2 Google pages, excluding duplicates, ads, and sites requiring payment or log-in. For each pathology, ChatGPT-4 Omni and OE were asked 2 standardized questions. Mean total DISCERN, reliability, and quality scores were calculated for each source and compared using 1-way analysis of variance (ANOVA) with post hoc Tukey tests.

Results

Each diagnosis yielded 10 to 17 hits and 1 response from ChatGPT and OE each. Mean total DISCERN scores were 35.9 for Google, 62.2 for ChatGPT, and 64.8 for OE (out of 80; Figure). One-way ANOVA revealed highly significant differences among groups (P <.0001). The Tukey test showed significant differences between Google vs ChatGPT (P <.001) and Google vs OE (P <.001) but not between ChatGPT and OE. Mean reliability scores were 19.9 for Google, 28.4 for ChatGPT, and 28.3 for OE; mean quality scores were 13.2, 30.5, and 31.1, respectively, both with significant differences between means (P <.0001). Significant differences were observed between Google and both LLMs for reliability (P <.0001) and quality (P <.0001), with no statistically significant difference between ChatGPT and OE for any score.

Conclusions

LLMs are a promising resource for patients seeking information about breast diagnoses, providing more accurate, higher-quality content than search engines. No significant differences were found between ChatGPT and OE, which suggests that open-access LLMs may be sufficient; however, given OE’s higher DISCERN score, specialized LLMs may still have added value. Although limited by few unblinded reviewers, these findings support integrating AI with tools like DISCERN to create reliable, accessible educational materials that enhance patient understanding without excessive medical jargon.