Rethinking and recomputing the value of machine learning models

作者：Burcu Sayin, Jie Yang, Xinyue Chen, Andrea Passerini, Fabio Casati · 发表于：Artificial Intelligence Review · 年份：2025 · DOI：10.1007/s10462-025-11242-6 · 被引用次数：5 · 研究领域：Adversarial Robustness in Machine Learning、Machine Learning and Algorithms、Imbalanced Data Classification Techniques

Abstract In this paper, we argue that the prevailing approach to training and evaluating machine learning models often fails to consider their real-world application within organizational or societal contexts, where they are intended to create beneficial value for people. We propose a shift in perspective, redefining model assessment and selection to emphasize integration into workflows that combine machine predictions with human expertise, particularly in scenarios requiring human intervention for low-confidence predictions. Traditional metrics like accuracy and f-score fail to capture the beneficial value of models in such hybrid settings. To address this, we introduce a simple yet theoretically sound “value” metric that incorporates task-specific costs for correct predictions, errors, and rejections, offering a practical framework for real-world evaluation. Through extensive experiments, we show that existing metrics fail to capture real-world needs, often leading to suboptimal choices in terms of value when used to rank classifiers. Furthermore, we emphasize the critical role of calibration in determining model value, showing that simple, well-calibrated models can often outperform more complex models that are challenging to calibrate.