Methodological insights into ChatGPT’s screening performance in systematic reviews

作者：Mahbod Issaiy, Hossein Ghanaati, Shahriar Kolahi, Madjid Shakiba, Amir Hossein Jalali, Diana Zarei, Sina Kazemian, Mahsa Alborzi Avanaki, Kavous Firouznia · 发表于：BMC Medical Research Methodology · 年份：2024 · DOI：10.1186/s12874-024-02203-8 · 被引用次数：73 · 研究领域：Artificial Intelligence in Healthcare and Education、Meta-analysis and systematic reviews、Radiomics and Machine Learning in Medical Imaging

BACKGROUND: The screening process for systematic reviews and meta-analyses in medical research is a labor-intensive and time-consuming task. While machine learning and deep learning have been applied to facilitate this process, these methods often require training data and user annotation. This study aims to assess the efficacy of ChatGPT, a large language model based on the Generative Pretrained Transformers (GPT) architecture, in automating the screening process for systematic reviews in radiology without the need for training data. METHODS: A prospective simulation study was conducted between May 2nd and 24th, 2023, comparing ChatGPT's performance in screening abstracts against that of general physicians (GPs). A total of 1198 abstracts across three subfields of radiology were evaluated. Metrics such as sensitivity, specificity, positive and negative predictive values (PPV and NPV), workload saving, and others were employed. Statistical analyses included the Kappa coefficient for inter-rater agreement, ROC curve plotting, AUC calculation, and bootstrapping for p-values and confidence intervals. RESULTS: ChatGPT completed the screening process within an hour, while GPs took an average of 7-10 days. The AI model achieved a sensitivity of 95% and an NPV of 99%, slightly outperforming the GPs' sensitive consensus (i.e., including records if at least one person includes them). It also exhibited remarkably low false negative counts and high workload savings, ranging from 40 to 8...