Matbench Discovery -- A framework to evaluate machine learning crystal stability predictions

作者：Janosh Riebesell, Rhys E. A. Goodall, Philipp Benner, Chiang, Yuan, Deng, Bowen, Ceder, Gerbrand, Asta, Mark, Alpha A. Lee, Anubhav Jain, Kristin A. Persson · 发表于：arXiv (Cornell University) · 年份：2023 · DOI：10.48550/arxiv.2308.14920 · 被引用次数：31 · 研究领域：Machine Learning in Materials Science、X-ray Diffraction in Crystallography、Computational Drug Discovery Methods

The rapid adoption of machine learning (ML) in domain sciences necessitates best practices and standardized benchmarking for performance evaluation. We present Matbench Discovery, an evaluation framework for ML energy models, applied as pre-filters for high-throughput searches of stable inorganic crystals. This framework addresses the disconnect between thermodynamic stability and formation energy, as well as retrospective vs. prospective benchmarking in materials discovery. We release a Python package to support model submissions and maintain an online leaderboard, offering insights into performance trade-offs. To identify the best-performing ML methodologies for materials discovery, we benchmarked various approaches, including random forests, graph neural networks (GNNs), one-shot predictors, iterative Bayesian optimizers, and universal interatomic potentials (UIP). Our initial results rank models by test set F1 scores for thermodynamic stability prediction: EquiformerV2 + DeNS > Orb > SevenNet > MACE > CHGNet > M3GNet > ALIGNN > MEGNet > CGCNN > CGCNN+P > Wrenformer > BOWSR > Voronoi fingerprint random forest. UIPs emerge as the top performers, achieving F1 scores of 0.57-0.82 and discovery acceleration factors (DAF) of up to 6x on the first 10k stable predictions compared to random selection. We also identify a misalignment between regression metrics and task-relevant classification metrics. Accurate regressors can yield high false-posi...