Susceptibility of Large Language Models to User-Driven Factors in Medical Queries

作者：Kyung Ho Lim, Ujin Kang, Xiang Li, Jin Sung Kim, Young‐Chul Jung, Sang Joon Park, Byung-Hoon Kim · 发表于：Journal of Healthcare Informatics Research · 年份：2025 · DOI：10.1007/s41666-025-00218-4 · 被引用次数：2 · 研究领域：Artificial Intelligence in Healthcare and Education、Machine Learning in Healthcare、Explainable Artificial Intelligence (XAI)

Large language models (LLMs) are increasingly used in healthcare; however, their reliability is shaped not only by model design but also by how queries are phrased and how complete the information is. This study assesses how user-driven factors, including misinformation framing, source authority, model personas, and omission of critical clinical details, influence the diagnostic accuracy and reliability of LLM-generated medical responses. Utilizing two public datasets (MedQA and Medbullets), we conducted two tests: (1) perturbation-evaluating LLM persona (assistant vs. expert AI), misinformation source authority (inexperienced vs. expert), and tone (assertive vs. hedged); and (2) ablation-omission of key clinical data. Proprietary LLMs (GPT-4o (OpenAI), Claude-3·5 Sonnet (Anthropic), Claude-3·5 Haiku (Anthropic), Gemini-1·5 Pro (Google), Gemini-1·5 Flash (Google)) and open-source LLMs (LLaMA-3 8B, LLaMA-3 Med42 8B, DeepSeek-R1 8B) were used for evaluation. Results show that in the perturbation test, all LLMs were susceptible to user-driven misinformation, with an assertive tone exerting the strongest overall impact, while proprietary models were more vulnerable to strong or authoritative misinformation. In the ablation test, omitting physical examination findings and laboratory results caused the largest accuracy decline. Proprietary models achieved higher baseline accuracy but demonstrated sharper performance drops under biased or incomplete input. These findings highlight t...