Diagnosis and Triage Performance of Contemporary Large Language Models on Short Clinical Vignettes

作者：Lei Xu, Wenzhe Zhao, Xin Huang · 发表于：Journal of Medical Systems · 年份：2025 · DOI：10.1007/s10916-025-02284-y · 被引用次数：6 · 研究领域：Artificial Intelligence in Healthcare and Education、Topic Modeling、Autopsy Techniques and Outcomes

General-purpose large language models (LLMs) are increasingly proposed for diagnostic and triage decision support, yet their reliability relative to humans remains unclear. We evaluated eight contemporary LLMs (ChatGPT-4, ChatGPT-o1, DeepSeek-V3, DeepSeek-R1, Gemini-2.0, Copilot, Grok-2, Llama-3.1) on 48 single-turn clinical vignettes spanning four triage levels (Emergent, 1-day, 1-week, Self-care). Models were tested without prompts and with structured prompts comprising exemplar cases. Primary outcomes were diagnostic and triage accuracy. Secondary measures included confusion matrices, over-triage, safety of advice, and the Capability Comparison Score (CCS). Structured prompting improved performance across models: mean diagnostic accuracy increased from 89.84% to 91.67%, and mean triage accuracy increased from 76.82% to 86.20%. The best diagnostic accuracy was 93.75% (ChatGPT-o1 and DeepSeek-R1; Grok-2 matched this when prompted). Prompting shifted models toward safety: safety of advice rose from 89.06% to 94.53%, accompanied by higher over-triage (from 53.15% to 65.62%). CCS values were numerically lower than accuracy but preserved rankings and conclusions (diagnosis CCS: from 49.54 to 50.46; triage CCS: from 47.66 to 52.34). Error analyses showed predominant over-triage, with rarer but clinically important under-triage. On concise, text-only vignettes, the diagnostic accuracy of advanced LLMs was high, in some cases nearing benchmarks set by physicians in prior studies, w...