Benchmarking four large language models’ performance of addressing Chinese patients' inquiries about dry eye disease: A two-phase study

作者：Runhan Shi, Steven Liu, Xinwei Xu, Zhengqiang Ye, Jin Yang, Qihua Le, Jini Qiu, Lijia Tian, Anji Wei, K. Y. Shan, Chen Zhao, Xinghuai Sun, Xingtao Zhou, Jiaxu Hong · 发表于：Heliyon · 年份：2024 · DOI：10.1016/j.heliyon.2024.e34391 · 被引用次数：20 · 研究领域：Ocular Surface and Contact Lens、Ocular Diseases and Behçet’s Syndrome、Ocular Infections and Treatments

Purpose To evaluate the performance of four large language models (LLMs)—GPT-4, PaLM 2, Qwen, and Baichuan 2—in generating responses to inquiries from Chinese patients about dry eye disease (DED). Design Two-phase study, including a cross-sectional test in the first phase and a real-world clinical assessment in the second phase. Subjects Eight board-certified ophthalmologists and 46 patients with DED. Methods The chatbots' responses to Chinese patients' inquiries about DED were assessed by the evaluation. In the first phase, six senior ophthalmologists subjectively rated the chatbots' responses using a 5-point Likert scale across five domains: correctness, completeness, readability, helpfulness, and safety. Objective readability analysis was performed using a Chinese readability analysis platform. In the second phase, 46 representative patients with DED asked the two language models (GPT-4 and Baichuan 2) that performed best in the in the first phase questions and then rated the answers for satisfaction and readability. Two senior ophthalmologists then assessed the responses across the five domains. Main outcome measures Subjective scores for the five domains and objective readability scores in the first phase. The patient satisfaction, readability scores, and subjective scores for the five-domains in the second phase. Results In the first phase, GPT-4 exhibited superior performance across the five domains (correctness: 4.47; completeness: 4.39; readability: 4.47; helpfulness...