A human-LLM collaborative annotation approach for screening articles on precision oncology randomized controlled trials

作者：Hui Chen, Jiale Zhao, Sheng Zheng, Xinyu Zhang, Huilong Duan, Xudong Lü · 发表于：BMC Medical Research Methodology · 年份：2025 · DOI：10.1186/s12874-025-02674-3 · 被引用次数：3 · 研究领域：Artificial Intelligence in Healthcare and Education、Meta-analysis and systematic reviews、Biomedical Text Mining and Ontologies

BACKGROUND: Supervised learning can accelerate article screening in systematic reviews, but still requires labor-intensive manual annotation. While large language models (LLMs) like GPT-3.5 offer a rapid and convenient alternative, their reliability is challenging. This study aims to design an efficient and reliable annotation method for article screening. METHODS: Given that relevant articles are typically a small subset of those retrieved articles during screening, we propose a human-LLM collaborative annotation method that focuses on verifying positive annotations made by the LLM. Initially, we optimized the prompt using a manually annotated standard dataset, refining it iteratively to achieve near-perfect recall for the LLM. Subsequently, the LLM, guided by the optimized prompt, annotated the articles, followed by human verification of the LLM-identified positive samples. This method was applied to screen articles on precision oncology randomized controlled trials, evaluating both its efficiency and reliability. RESULTS: For prompt optimization, a standard dataset of 200 manually annotated articles was equally divided into a tuning set and a validation set (1:1 ratio). Through iterative prompt optimization, the LLM achieved near-perfect recall in the tuning and validation sets, with 100% and 85.71%, respectively. Using the optimized prompt, we conducted collaborative annotation. To evaluate its performance, we manually reviewed a random sample of 300 articles that had bee...