Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionals

作者：Kathrin Blagec, Jakob Kraiger, Wolfgang Frühwirt, Matthias Samwald · 发表于：arXiv (Cornell University) · 年份：2022 · DOI：10.48550/arxiv.2201.07040 · 被引用次数：1 · 研究领域：Artificial Intelligence in Healthcare and Education、Machine Learning in Healthcare、Radiomics and Machine Learning in Medical Imaging

Publicly accessible benchmarks that allow for assessing and comparing model performances are important drivers of progress in artificial intelligence (AI). While recent advances in AI capabilities hold the potential to transform medical practice by assisting and augmenting the cognitive processes of healthcare professionals, the coverage of clinically relevant tasks by AI benchmarks is largely unclear. Furthermore, there is a lack of systematized meta-information that allows clinical AI researchers to quickly determine accessibility, scope, content and other characteristics of datasets and benchmark datasets relevant to the clinical domain. To address these issues, we curated and released a comprehensive catalogue of datasets and benchmarks pertaining to the broad domain of clinical and biomedical natural language processing (NLP), based on a systematic review of literature and online resources. A total of 450 NLP datasets were manually systematized and annotated with rich metadata, such as targeted tasks, clinical applicability, data types, performance metrics, accessibility and licensing information, and availability of data splits. We then compared tasks covered by AI benchmark datasets with relevant tasks that medical practitioners reported as highly desirable targets for automation in a previous empirical study. Our analysis indicates that AI benchmarks of direct clinical relevance are scarce and fail to cover most work activities that clinicians want to see addressed. I...