Reproducibility and Robustness of Localized Mortality Prediction

作者：Evelyn Nitch-Griffin, Amy Peterson, Yara Skaf, Jason Cory Brunson · 发表于：medRxiv · 年份：2024 · DOI：10.1101/2024.06.04.24308417 · 被引用次数：1 · 研究领域：Insurance, Mortality, Demography, Risk Management

Abstract Background While localized modeling—the use of predictive models to perform the adaptation step in case-based reasoning—has been evaluated in several experimental settings, its reported successes have infrequently been independently and externally validated. Objective We aimed to extend and validate an experimental study of mortality prediction in a critical care population and to assess the importance of several methodological factors to predictive performance. Methods We reproduced the workflow of Lee, Maslove, and Dubin (2015) using an updated database. We evaluated performance as area under the receiver operating characteristic curve and under the precision–recall curve and calibration as weakness of evidence in Hosmer– Lemeshow tests. We compared the effects of several modeling choices, including how relevance is quantified, and how relevance cohorts are retrieved, and the choice of model. We compared ours to previous results and used linear regression to quantify the role of each modeling choice on performance. Results Overall performance and its relationship to cohort size validated previous results. These relationships varied by model family as expected, though we observed no advantage of decision trees over a model-free approach and poor performance by random forests. An alternate choice of unlearned similarity measure yielded marginal and inconsistent performance differences. Denominating cohorts by similarity threshold rather than by cardinality yielded ma...