Predicting the shear modulus of a crystal

The shear modulus says how strongly a solid resists being twisted, and it is one of the elastic constants most often screened for in structural materials. This task asks for it from the crystal structure, on 10,987 compounds with computed elastic tensors.

Task

Given a crystal structure, predict $\log_{10}$ of the Voigt-Reuss-Hill average shear modulus in GPa. Scoring is the mean absolute error of that logarithm over the five fixed folds; the log transform makes the metric effectively a relative error on the modulus itself.

Background

The elastic tensors come from de Jong and co-workers' systematic DFT survey, filtered by Matbench to drop entries far from the hull, physically inconsistent tensors and noble-gas compounds. Together with the bulk-modulus task it forms the suite's mid-sized pair of elastic problems, large enough for graph networks to lead. Shear is the harder of the two: the same leader reaches 0.0491 on the bulk modulus.

Data and evaluation

• Data: matbench_log_gvrh.json.gz from MPContribs-ML (10,987 records, each a pymatgen Structure and a log10 shear modulus in GPa), at https://ml.materialsproject.org/projects/matbench_log_gvrh or via task.load() in the matbench package; the elastic tensors are from de Jong and co-workers.
• Split: five folds fixed in matbench_v0.1_validation.json, built with scikit-learn's KFold(n_splits=5, shuffle=True, random_state=18012019). Train on each fold's training set, tune inside it only, predict its test set, report the mean. Changing the folds invalidates the comparison.
• Metric: mean absolute error in log10(GPa) over the five test folds, lower is better. The leaderboard also lists the fold-to-fold standard deviation, RMSE and worst single-sample error.
• Official evaluation code: the matbench package (MIT), https://github.com/materialsproject/matbench; the task's record(fold, predictions), as in mb.matbench_log_gvrh.record(...) on a MatbenchBenchmark mb, scores a fold and mb.to_file(...) writes the submission file.

What is known

• Top of the task leaderboard: coNGN, mean MAE 0.0670 in log10(GPa) (standard deviation 0.0006).
• Runners-up: coGN 0.0689, ALIGNN 0.0715 and MODNet v0.1.12 0.0731.
• A dummy regressor scores 0.2931, the target's mean absolute deviation, so the best models cut the error more than fourfold.
• Fold-to-fold variation is tiny here (standard deviation below 0.001), so even a 0.002 improvement is measurable.
• Unmerged claim: pull request #347 (April 2024) submits DensGNN, whose results file records a mean MAE of 0.0664; it has not been reviewed or added to the leaderboard.
• The leaderboard's repository has merged no submission since 20 January 2024; newer ones wait as open pull requests, so the official numbers may lag what has been achieved.

What would count as progress

• A mean MAE below 0.067 log10(GPa) on the five fixed folds, beating coNGN, submitted as the matbench benchmark file so every fold's predictions can be re-scored.
• Code and a pinned environment that reproduce the run from the fixed folds.
• Per-fold numbers, not just the mean, so that a gain can be compared with the leader fold by fold on the same folds; a small gain should hold on each fold or over repeated training runs. Reproducing the current leader counts too, since the board has not moved since 2024.

References

1. M. de Jong et al., "Charting the complete elastic properties of inorganic crystalline compounds", Scientific Data 2, 150009 (2015). https://doi.org/10.1038/sdata.2015.9
2. A. Dunn, Q. Wang, A. Ganose, D. Dopp and A. Jain, "Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm", npj Computational Materials 6, 138 (2020). https://doi.org/10.1038/s41524-020-00406-3
3. R. Ruff, P. Reiser, J. Stühmer and P. Friederich, "Connectivity optimized nested line graph networks for crystal structures", Digital Discovery 3, 594–601 (2024). https://doi.org/10.1039/D4DD00018H
4. P.-P. De Breuck, M. L. Evans and G.-M. Rignanese, "Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet", Journal of Physics: Condensed Matter 33, 404002 (2021). https://doi.org/10.1088/1361-648X/ac1280

Source

Imported from Matbench v0.1 matbench_log_gvrh, curated by Materials Project (Dunn et al.). Open according to the source as of 2026-10-20. The statement above is written in our own words; see the source for the original wording, history and discussion.