Toward expert-level medical text validation with language models

作者：Asad Aali, Vasiliki Bikia, Maya Varma, Nicole Chiou, Sophie Ostmeier, Arnav Singhvi, Magdalini Paschali, Ashwin Kumar, Andrew Johnston, Karimar Amador-Martinez, Eduardo Guerrero, Paola Naovi Cruz Rivera, Sergios Gatidis, Christian Bluethgen, Eduardo Pontes Reis, Eddy D. Zandee van Rilland, Poonam Hosamani, Kevin Keet, Minjoung Go, Evelyn Ling, David B. Larson, Curtis P. Langlotz, Roxana Daneshjou, Jason Hom, Sanmi Koyejo, Emily Alsentzer, Akshay Chaudhari · 发表于：npj Digital Medicine · 年份：2026 · DOI：10.1038/s41746-026-03084-5 · 研究领域：Topic Modeling、Sentiment Analysis and Opinion Mining、Biomedical Text Mining and Ontologies

With the growing use of language models (LMs) in clinical environments, there is an immediate need to evaluate the accuracy of LMs. Detecting errors in LM-generated text is challenging because (1) manual review is costly and (2) expert-composed reference outputs are often unavailable in real-world settings. While the “LLM-as-a-judge” paradigm (a LM evaluating another LM) offers scalable evaluation, even frontier LMs can miss subtle but clinically significant errors. To address these challenges, we propose MedVAL, a novel, self-supervised, data-efficient distillation method that leverages synthetic data to train evaluator LMs to assess whether LM-generated medical outputs are factually consistent with inputs, without requiring physician labels or reference outputs. To evaluate LM performance, we introduce MedVAL-Bench, a dataset of 840 physician-annotated outputs across 6 diverse clinical use cases capturing real-world challenges. Each output is reviewed following a physician-defined taxonomy of risk levels and error categories, enabling evaluation of LMs in making deployment safety decisions. Across 10 state-of-the-art LMs spanning open-source, proprietary, and medically adapted models, MedVAL distillation significantly improves ( p < 0.001) alignment with physicians across seen and unseen tasks, increasing average F1 scores from 66% to 83%. Despite strong baseline performance, MedVAL improves the best-performing proprietary LM (GPT-4o) by 8% without training on physician-lab...