Multimodal large language models in ultrasound diagnosis of breast masses: a multicenter comparative analysis based on GPT-4o, radiologists, and convolutional neural network (CNN)

作者：Jiaqian Yao, Rui Zhang, Ze-Bang Yang, Bo Zhang, Shuangquan Jiang, Lin Jiang, Xiaoer Zhang, Xiaoyan Xie, Tongyi Huang, Minghui Xu · 发表于：Quantitative Imaging in Medicine and Surgery · 年份：2025 · DOI：10.21037/qims-2025-92 · 被引用次数：3 · 研究领域：Artificial Intelligence in Healthcare and Education、Radiomics and Machine Learning in Medical Imaging、AI in cancer detection

Background: An advanced version of large language model (LLM), ChatGPT 4o (GPT-4o), has shown capacity in image-text pair interpretation, yet the performance in medical image analysis remains unclear. This study aimed to evaluate the diagnostic capacity of GPT-4o in breast ultrasound (US) datasets. Methods: US exams including breast images and original reports were respectively included from January 2021 to December 2023 in three hospitals throughout China. The diagnostic performance in distinguishing benign or malignant breast masses of GPT-4o was assessed through two approaches: image-strategy and image-combined-text-strategy. Fleiss kappa was calculated to determine intra-LLM consistency. Thereafter, diagnostic accuracy was evaluated and compared with the convolutional neural network (CNN) model and 95 human experts with various levels of expertise from 60 institutions in China. Responses from GPT-4o were rated by diagnostic confidence and radiologist’s evaluation. Results: The observations of 80 breast masses (37 malignant, 43 benign) from 80 patients [median age, 42.5 years; interquartile range (IQR), 37.0–53.0 years] were enrolled. GPT-4o with image-strategy exhibited a fair consistency [0.25, 95% confidence interval (CI): 0.07–0.43], whereas the agreement of image-combined-text-strategy was excellent (0.81, 95% CI: 0.67–0.91). Diagnostic accuracy improved when deploying the image-combined-text-strategy compared to only image [58% (46 of 80) vs. 70% (56 of 80), P=0.031]...