Predicting bulk metallic glass formation from composition

Some alloys freeze into a glass instead of a crystal, which gives them unusual strength and elasticity, but which compositions do so is still mostly empirical. This task asks for that call from the composition alone, on 5,680 reported ternary alloys.

Task

Given a chemical composition as a formula string, predict whether it is fully glass-forming, that is whether it yields an amorphous phase (label 1) or not (label 0). Scoring is the mean ROC-AUC over the five fixed stratified folds. The classes are skewed: 71.04% of the entries are glass-forming.

Background

The labels were read off the Landolt-Börnstein volume on nonequilibrium phase diagrams of ternary amorphous alloys, and deduplicated so that no composition appears with both labels. Glass-forming ability depends on cooling rate and on competing crystalline phases, so the labels record what a particular experimental route produced rather than a thermodynamic property; a model can only learn the composition-level regularities. It is the one task in the suite that touches amorphous materials.

Data and evaluation

Data: matbench_glass.json.gz from MPContribs-ML (5,680 records, each a formula string and a binary glass-forming-ability label), at https://ml.materialsproject.org/projects/matbench_glass or via task.load() in the matbench package; the labels come from the Landolt-Börnstein compilation by Kawazoe and co-workers.
Split: five folds fixed in matbench_v0.1_validation.json, built with scikit-learn's StratifiedKFold(n_splits=5, shuffle=True, random_state=18012019). Train on each fold's training set, tune inside it only, predict its test set, report the mean. Changing the folds invalidates the comparison.
Metric: mean ROC-AUC over the five test folds, higher is better. Matbench thresholds probabilities at 0.5 before scoring, so the reported ROC-AUC equals the balanced accuracy of the hard labels; an unmerged fix (#245, 2023) would score the probabilities instead.
Official evaluation code: the matbench package (MIT), https://github.com/materialsproject/matbench; the task's record(fold, predictions), as in mb.matbench_glass.record(...) on a MatbenchBenchmark mb, scores a fold and mb.to_file(...) writes the submission file.

What is known

Top of the task leaderboard: MODNet v0.1.12, mean ROC-AUC 0.9603 (standard deviation 0.0075), and it is also the best general-purpose entry.
Runners-up: AMMExpress v2020 at 0.8607 and RF-SCM/Magpie at 0.8587, a full ten points behind; an unmerged gradient-boosting entry (#363, July 2026) reports 0.8658.
A dummy classifier scores 0.5005.
The gap to second place is unusually wide for this suite, so reproducing MODNet's result independently is itself useful.
The leaderboard's repository has merged no submission since 20 January 2024; newer ones wait as open pull requests, so the official numbers may lag what has been achieved.

What would count as progress

A mean ROC-AUC above 0.9603 on the five fixed folds, beating MODNet v0.1.12, submitted as the matbench benchmark file so every fold's predictions can be re-scored.
Code and a pinned environment that reproduce the run from the fixed folds.
Per-fold numbers, not just the mean, so that a gain can be compared with the leader fold by fold on the same folds; a small gain should hold on each fold or over repeated training runs. Reproducing the current leader counts too, since the board has not moved since 2024.

References

Y. Kawazoe, J.-Z. Yu, A.-P. Tsai and T. Masumoto (eds.), Nonequilibrium Phase Diagrams of Ternary Amorphous Alloys, Landolt-Börnstein Group III Condensed Matter 37A, Springer (1997). https://link.springer.com/book/10.1007/b58222
A. Dunn, Q. Wang, A. Ganose, D. Dopp and A. Jain, "Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm", npj Computational Materials 6, 138 (2020). https://doi.org/10.1038/s41524-020-00406-3
P.-P. De Breuck, M. L. Evans and G.-M. Rignanese, "Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet", Journal of Physics: Condensed Matter 33, 404002 (2021). https://doi.org/10.1088/1361-648X/ac1280

Source

Imported from Matbench v0.1 matbench_glass, curated by Materials Project (Dunn et al.). Open according to the source as of 2026-10-20. The statement above is written in our own words; see the source for the original wording, history and discussion.