Potential of multimodal large language models for data mining of medical images and free-text reports

作者：Yutong Zhang, Yi Pan, Tianyang Zhong, Peixin Dong, Kangni Xie, Yuxiao Liu, Hanqi Jiang, Zihao Wu, Zhengliang Liu, Wei Zhao, Wei Zhang, Shijie Zhao, Tuo Zhang, Xi Jiang, Dinggang Shen, Tianming Liu, Xin Zhang · 发表于：Meta-Radiology · 年份：2024 · DOI：10.1016/j.metrad.2024.100103 · 被引用次数：44 · 研究领域：COVID-19 diagnosis using AI、AI in cancer detection、Radiomics and Machine Learning in Medical Imaging

Medical images and radiology reports are essential for physicians to diagnose medical conditions . However, the vast diversity and cross-source heterogeneity inherent in these data have posed significant challenges to the generalizability of current data-mining methods for clinical decision-making. Recently, multimodal large language models (MLLMs), especially Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models, have revolutionized numerous domains, significantly impacting the medical field. In this study, we conducted a detailed evaluation of the performance of the Gemini series models (including Gemini-1.0-Pro-Vision, Gemini-1.5-Pro, and Gemini-1.5-Flash) and GPT series models (including GPT-4o, GPT-4-Turbo, and GPT-3.5-Turbo) across 14 medical datasets, covering 5 medical imaging categories (dermatology, radiology, dentistry , ophthalmology , and endoscopy) and 3 radiology report datasets. The investigated tasks encompass disease classification , lesion segmentation , anatomical localization, disease diagnosis, report generation, and lesion detection. Moreover, we also validated the performance of the Claude-3-Opus, Yi-Large, Yi-Large-Turbo, and LLaMA 3 models to gain a comprehensive understanding of the MLLM models in the medical field. Our experimental results demonstrated that Gemini-series models excelled in report generation and lesion detection but faces challenges in disease classification and anatomical localization. Conversely, GPT-series models exhibite...