Performance analysis of large language models in multi-disease detection from chest computed tomography reports: a comparative study

作者：Peng Luo, Chaofan Fan, Anghua Li, Tong Jiang, Aimin Jiang, Qi Chang, Wenyi Gan, Lingxuan Zhu, Weiming Mou, Dongqiang Zeng, Bufu Tang, Mingjia Xiao, Guangdi Chu, Zhenyu Liang, Junyi Shen, Zaoqu Liu, Ting Wei, Quan Cheng, Anqi Lin, Xin Chen · 发表于：International Journal of Surgery · 年份：2025 · DOI：10.1097/js9.0000000000002582 · 被引用次数：4 · 研究领域：COVID-19 diagnosis using AI、Artificial Intelligence in Healthcare and Education、Radiology practices and education

BACKGROUND: Computed tomography (CT) is widely acknowledged as the gold standard for diagnosing thoracic diseases. However, the accuracy of interpretation significantly depends on radiologists' expertise. Large language models (LLMs) have shown considerable promise in various medical applications, particularly in radiology. This study aims to assess the performance of leading LLMs in analyzing unstructured chest CT reports and to examine how different questioning methodologies and fine-tuning strategies influence their effectiveness in enhancing chest CT diagnosis. MATERIALS AND METHODS: This retrospective analysis evaluated 13 489 chest CT reports encompassing 13 common thoracic conditions across pulmonary, cardiovascular, pleural, and upper abdominal systems. Five LLMs (Claude-3.5-Sonnet, GPT-4, GPT-3.5-Turbo, Gemini-Pro, Qwen-Max) were assessed using dual questioning methodologies: multiple-choice and open-ended. Radiologist-curated datasets underwent rigorous preprocessing, including RadLex terminology standardization, multi-step diagnostic validation, and exclusion of ambiguous cases. Model performance was quantified via Subjective Answer Accuracy Rate (SAAR), Reference Answer Accuracy Rate (RAAR), and Area Under the Receiver Operating Characteristic (ROC) Curve analysis. GPT-3.5-Turbo underwent fine-tuning (100 iterations with one training epoch) on 200 high-performing cases to enhance diagnostic precision for initially misclassified conditions. RESULTS: GPT-4 demonstra...