A multi-modal deep learning solution for precise pneumonia diagnosis: the PneumoFusion-Net model

作者：Yujie Wang, Can Liu, Yu-Yan Fan, Chenyue Niu, Wan‐Yun Huang, Ying Pan, Jingze Li, Jingze Li, Yilin Wang, Jun Li, Jun Li · 发表于：Frontiers in Physiology · 年份：2025 · DOI：10.3389/fphys.2025.1512835 · 被引用次数：11 · 研究领域：COVID-19 diagnosis using AI、Pneumonia and Respiratory Infections、Domain Adaptation and Few-Shot Learning

Background: Pneumonia is considered one of the most important causes of morbidity and mortality in the world. Bacterial and viral pneumonia share many similar clinical features, thus making diagnosis a challenging task. Traditional diagnostic method developments mainly rely on radiological imaging and require a certain degree of consulting clinical experience, which can be inefficient and inconsistent. Deep learning for the classification of pneumonia in multiple modalities, especially integrating multiple data, has not been well explored. Methods: The study introduce the PneumoFusion-Net, a deep learning-based multimodal framework that incorporates CT images, clinical text, numerical lab test results, and radiology reports for improved diagnosis. In the experiments, a dataset of 10,095 pneumonia CT images was used-including associated clinical data-most of which was used for training and validation while keeping part of it for validation on a held-out test set. Five-fold cross-validation was considered in order to evaluate this model, calculating different metrics including accuracy and F1-Score. Results: PneumoFusion-Net, which achieved 98.96% classification accuracy with a 98% F1-score on the held-out test set, is highly effective in distinguishing bacterial from viral types of pneumonia. This has been highly beneficial for diagnosis, reducing misdiagnosis and further improving homogeneity across various data sets from multiple patients. Conclusion: PneumoFusion-Net offers...