Predicting formation energy from crystal structure

Formation energy says how much energy is released when a compound is made from its elements, and it is the quantity screening pipelines use to decide whether a hypothetical crystal is worth a DFT run. This task asks for it from the structure, on 132,752 Materials Project entries.

Task

Given a crystal structure, predict the formation energy in electronvolts per atom as computed by the Materials Project. Scoring is the mean absolute error over the five fixed folds. This is the largest task in the suite and the one on which crystal graph networks are usually compared.

Background

The labels are Materials Project formation energies, filtered by Matbench to drop entries above 2.5 eV and noble-gas compounds. Formation energy is the easiest of the suite's targets, because it is close to a sum of local bonding contributions that a message-passing network can learn; the leader is now at 17 meV per atom. The remaining questions are about generalisation rather than fit, which is what the separate Matbench Discovery benchmark was built to probe.

Data and evaluation

• Data: matbench_mp_e_form.json.gz from MPContribs-ML (132,752 records, each a pymatgen Structure and a formation energy in eV/atom), at https://ml.materialsproject.org/projects/matbench_mp_e_form or via task.load() in the matbench package; the energies are Materials Project calculations.
• Split: five folds fixed in matbench_v0.1_validation.json, built with scikit-learn's KFold(n_splits=5, shuffle=True, random_state=18012019). Train on each fold's training set, tune inside it only, predict its test set, report the mean. Changing the folds invalidates the comparison.
• Metric: mean absolute error in eV/atom over the five test folds, lower is better. The leaderboard also lists the fold-to-fold standard deviation, RMSE and worst single-sample error.
• Official evaluation code: the matbench package (MIT), https://github.com/materialsproject/matbench; the task's record(fold, predictions), as in mb.matbench_mp_e_form.record(...) on a MatbenchBenchmark mb, scores a fold and mb.to_file(...) writes the submission file.

What is known

• Top of the task leaderboard: coGN, mean MAE 0.0170 eV/atom (standard deviation 0.0003).
• Runners-up: coNGN 0.0178, ALIGNN 0.0215 and SchNet (kgcnn v2.1.0) 0.0218 eV/atom.
• A dummy regressor scores 1.0059 eV/atom, the target's mean absolute deviation, so the leaders are about sixty times better.
• With 17 algorithms scored besides the dummy this is the most contested task in the suite, and the top four are within 5 meV/atom of each other.
• Unmerged claims: pull request #322 (February 2024) submits iComFormer and eComFormer, whose results files record 0.0165 eV/atom for both, and #361 (March 2026) reports 0.0169 eV/atom for frozen MACE-MH-1 descriptors with a small neural-network head. Neither has been reviewed or added to the leaderboard.
• The leaderboard's repository has merged no submission since 20 January 2024; newer ones wait as open pull requests, so the official numbers may lag what has been achieved.

What would count as progress

• A mean MAE below 0.017 eV/atom on the five fixed folds, beating coGN, submitted as the matbench benchmark file so every fold's predictions can be re-scored.
• Code and a pinned environment that reproduce the run from the fixed folds.
• Per-fold numbers, not just the mean, so that a gain can be compared with the leader fold by fold on the same folds; a small gain should hold on each fold or over repeated training runs. Reproducing the current leader counts too, since the board has not moved since 2024.

References

1. A. Jain et al., "The Materials Project: A materials genome approach to accelerating materials innovation", APL Materials 1, 011002 (2013). https://doi.org/10.1063/1.4812323
2. A. Dunn, Q. Wang, A. Ganose, D. Dopp and A. Jain, "Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm", npj Computational Materials 6, 138 (2020). https://doi.org/10.1038/s41524-020-00406-3
3. R. Ruff, P. Reiser, J. Stühmer and P. Friederich, "Connectivity optimized nested line graph networks for crystal structures", Digital Discovery 3, 594–601 (2024). https://doi.org/10.1039/D4DD00018H

Source

Imported from Matbench v0.1 matbench_mp_e_form, curated by Materials Project (Dunn et al.). Open according to the source as of 2026-10-20. The statement above is written in our own words; see the source for the original wording, history and discussion.