LLM Performance in PMOS/PCOS Information: DeepSeek-V4 Outperforms GPT-5.5 and Gemini 3.5 in Clinician-Rated Quality and Empathy
What the Researchers Tested
This study employed a comparative observational design to assess the output quality of three distinct, free-access large language models (LLMs): DeepSeek-V4, GPT-5.5, and Gemini 3.5. A standardized set of 33 clinician-developed questions pertaining to polyendocrine metabolic ovarian syndrome (PMOS, previously PCOS) was submitted once to each LLM during a specific period (July 20-25, 2026). The questions were designed to cover a broad spectrum of clinical information relevant to PMOS/PCOS. The core evaluation involved two independent clinicians who blindly rated each of the 99 generated responses. Key evaluation criteria included information quality (using the DISCERN instrument, which assesses reliability and quality of health information), transparency (evaluated via EQIP guidelines), adherence to JAMA benchmarks for health information, overall global quality, and perceived empathy. Six distinct readability indices (e.g., ARI, GFI, FKGL, SMOG, Coleman-Liau, FRES) were also calculated for each response to provide an objective measure of textual complexity. Statistical analysis, including Friedman tests and paired signed-rank permutation comparisons with Holm correction, was applied to identify significant differences across the LLM configurations.
What the Study Found
DeepSeek-V4 consistently achieved higher final clinician ratings compared to GPT-5.5 and Gemini 3.5 across all five clinician-rated measures (information quality, transparency, JAMA benchmarks, global quality, and perceived empathy), with Holm-adjusted pairwise p < 0.05.
Median DISCERN scores for information quality were 67.0 for DeepSeek-V4, 62.0 for GPT-5.5, and 61.0 for Gemini 3.5, indicating DeepSeek-V4 provided more reliable health information.
DeepSeek-V4 demonstrated higher median values for ARI, GFI, FKGL, and SMOG readability indices, suggesting its responses were generally more complex or required a higher reading level.
GPT-5.5 had the highest median Coleman-Liau score, another indicator of text complexity, while the FRES index showed no significant overall difference across models (adjusted p = 0.100).
Initial inter-rater agreement for perceived empathy was low (ICC 0.05-0.43), indicating subjectivity in empathy assessment, though DeepSeek-V4 ultimately received higher clinician empathy ratings.
Why This Matters to Clinical Practice
The increasing reliance on freely accessible large language models by both patients and healthcare professionals for health-related information necessitates rigorous evaluation of their outputs. This study directly addresses the critical need to understand the characteristics and reliability of LLM-generated medical information, particularly concerning complex conditions like PMOS/PCOS. Unreliable, unclear, or unempathetic responses can lead to patient misinformation, anxiety, and potentially detrimental health decisions, or inefficient clinical workflows if healthcare providers spend excessive time verifying AI-generated data. For practicing clinicians, the findings provide a preliminary guide on which LLM configurations may offer higher-quality and more empathetic information, although the study cautions against establishing stable model rankings. Understanding these differences is crucial for integrating AI tools responsibly into clinical practice, patient education, and even administrative tasks. The variability in readability also highlights the need for LLMs to adapt their output for diverse audiences, from laypersons seeking initial understanding to specialists requiring in-depth, nuanced information. While not establishing clinical correctness, the insights into information quality and transparency are foundational for future clinical applications.
Clinical & Workflow Takeaways
Clinicians and department heads considering the integration of LLMs for generating patient-facing information or for internal clinical support should be aware of the substantial variability in output quality, transparency, and perceived empathy across different models. While DeepSeek-V4 showed superior performance in this specific evaluation, relying solely on any single LLM for critical medical information without human oversight is not advisable. Clinical teams should establish rigorous internal validation processes for AI-generated content. For patient education materials, clinicians may need to cross-reference multiple LLM outputs and extensively edit for clarity, accuracy, and appropriate tone, especially given the observed differences in readability scores across models. The study suggests that for information pertaining to PMOS/PCOS, DeepSeek-V4 might be a stronger starting point due to higher clinician-rated quality.
Limitations and What Remains Uncertain
The study's limitations include the evaluation of only a specific set of 33 clinician-formulated questions about PMOS/PCOS, which are 'treatment-heavy' and may not generalize to other medical domains or question types. The findings are specific to the configurations of DeepSeek-V4, GPT-5.5, and Gemini 3.5 available during July 20-25, 2026, and LLM capabilities evolve rapidly, meaning these rankings may not be stable over time. Crucially, the study explicitly states that it does not establish clinical correctness, safety, or effectiveness for patients, focusing instead on communication characteristics. The low initial inter-rater reliability for perceived empathy (ICC 0.05-0.43) highlights the subjective nature and potential challenges in consistently measuring this attribute. The use of only two clinician raters, while experienced, could introduce some bias, and larger, multi-specialty clinician panels might offer a more robust evaluation.
Explore the Work Behind Clinical AI
MedXchange Research Fellowship
Publish Clinical AI Research With Us
Our 10-week fellowship mentors physicians and researchers to build foundation models, analyze multimodal health datasets, and co-author peer-reviewed clinical studies.
Related Clinical AI Stories
LLMs for Atrial Fibrillation Patient Education: Cardiologist vs. Non-Medical Professional Perceptions of ChatGPT and Google Gemini Quality
A recent pilot study published in JMIR Formative Research comparatively analyzed the quality of atrial fibrillation (AF) explanations generated by ChatGPT and Google Gemini, specifically tailored for patient education. The study aimed to assess the overall quality of these AI platforms, compare their responses, and identify differences in perception between cardiologists and non-medical professionals. The prompt, "Explain atrial fibrillation and how to treat it to a patient," was used to generate responses from both AI models. These responses were then evaluated by a cohort of 28 participants, comprising cardiologists/cardiology fellows and non-medical college graduates, using a survey derived from the Patient Education Materials Assessment Tool (PEMAT) and DISCERN criteria, rated on a 5-point Likert scale. The findings indicated no significant overall quality difference between ChatGPT and Google Gemini for AF education. However, crucial discrepancies emerged in how different participant groups evaluated specific aspects. Cardiologists rated AI responses higher for perceived bias and for explaining the consequences of untreated AF, while non-medical professionals favored structural and organizational features such as visual cues, informative headers, concise sections, identifiable patient actions, and references to additional resources. This suggests that while both platforms are viable, their utility in clinical settings requires careful consideration of the target audience and a balanced evaluation by specialists.
AI Chatbot Performance in Orthopedic Education: GPT-4o Outperforms AAOS Members in Adult Reconstruction
A recent study published in the Rhode Island Medical Journal evaluated the performance of seven contemporary AI chatbots on 55 multiple-choice questions from the AAOS ResStudy question bank, focusing on adult reconstruction. The objective was to assess the accuracy, response time, and citation reliability of these AI models when confronted with clinically oriented orthopedic questions. Overall, the chatbots achieved a moderate accuracy of 67.5%, with significant variability among models and question types. Notably, GPT-4o demonstrated the highest accuracy at 80%, surpassing the average performance of AAOS members, who scored 75% on the same questions. Co-Pilot secured the second-best performance at 70.9%. The study meticulously categorized questions by anatomic region, concept (clinical presentation, prognosis, management), and AAOS-defined difficulty, revealing that AI chatbots excelled in descriptive knowledge (Clinical Presentation) but struggled with complex applied reasoning, such as prognosis and management. The findings suggest a valuable supplementary role for AI in orthopedic education, while highlighting areas needing further development.
LLM Chatbot Offers Promising, Yet Flawed, Allergen Avoidance Guidance in Contact Dermatitis
A recent quantitative evaluation study explored the utility of a Large Language Model (LLM) chatbot, specifically GPT-4, in providing allergen-free product recommendations for patients with contact dermatitis. The study assessed the LLM's ability to recommend products free of 30 common allergen groups found in international standard patch tests. While the chatbot demonstrated a high proportion of valid responses, particularly for cosmetics and medications, clinicians must recognize its limitations, especially concerning complex allergens prone to cross-reactivity. Over a period from September 2023 to January 2024, GPT-4 was prompted to suggest five allergen-free products for each allergen group, utilizing three distinct question formats. Dermatologists manually reviewed and categorized 2250 responses for validity, invalidity, cross-reactions, unknown status, and unresponsiveness. The findings indicate a significant potential for LLM tools to assist in allergen avoidance counseling, albeit with a crucial need for human verification and oversight.