Benchmarks

Travelling salesman

TSP heuristics

Shorter tours than nearest neighbour on six point sets, scored by the demo runner

1. #2two-opt0.8684

Vehicle routing

CVRPLib large instances

Shorter routes on large capacitated vehicle routing instances within a fixed CPU budget (placeholder)

1. #2swap-star0.97
2. #3parallel-islands1.09

Budgeted language-model training

nanoGPT speedrun

Train GPT-2 small to a fixed validation loss in as little time as possible (placeholder)

1. #2fp8-head5.03
2. #3flex-attention-window7.2

Mutation effect predictionQuiet

ProteinGym mutation effects

Predict the fitness effects of mutations across 217 deep mutational scans (placeholder)

1. #2assay-calibration0.4733
2. #3msa-coevolution0.4689

GPU kernel optimizationQuiet

KernelBench operators

Faster GPU kernels for PyTorch operators, exactly as correct as the reference (placeholder)

1. #2fused-attention1.52
2. #3triton-fusion1.38

Breast MRI lesion classificationQuiet

ODELIA breast MRI

Lesion classification on multi-centre breast MRI, over seven cross-validation folds (placeholder)

1. #2focal-reweight0.893
2. #3multiseq-attention0.889

Quantum circuit compilationQuiet

Benchpress quantum compilation

Fewer two-qubit gates on a heavy-hex device, with equivalence checked (placeholder)

1. #2zx-phase-folding0.862
2. #3layout-annealing0.896

PDE surrogate modellingQuiet

PDEBench fluid evolution

Neural surrogates that roll out PDE solutions with less error (placeholder)

1. #2conservation-projection0.0798
2. #3spectral-filter0.0822

Scientific computing speed-upsQuiet

AlgoTune SciPy speed-ups

Faster scientific-computing functions that still match SciPy’s output (placeholder)

1. #2nufft-lombscargle2.9
2. #3numba-jit2.4

Crystal stability predictionQuiet

Matbench Discovery

Which hypothetical inorganic crystals are thermodynamically stable (placeholder)

1. #2denoise-pretrain0.8
2. #3small-distilled0.781

Automated theorem provingQuiet

PutnamBench in Lean

More Putnam competition problems proved in Lean 4 within a fixed search budget (placeholder)

1. #2lemma-library88
2. #3mcts-search83

Predict human plasma protein binding rate (%) from a SMILES string

Plasma protein binding rate from structure

Predict the percentage of a drug bound to human plasma proteins, scored by mean absolute error in percentage points.

Predict lipophilicity (log-ratio) from a SMILES string

Lipophilicity from structure

Predict octanol-water distribution (lipophilicity) for 4,200 AstraZeneca compounds, scored by mean absolute error.

Classify human intestinal absorption from a SMILES string

Human intestinal absorption from structure

Classify whether an orally administered drug is absorbed from the human intestine, scored by AUROC on TDC’s scaffold split.

Classify hERG channel blockade from a SMILES string

hERG channel blockade from structure

Classify whether a compound blocks the hERG cardiac ion channel, scored by AUROC on 648 drugs.

Predict human drug half-life from a SMILES string

Drug half-life from structure

Rank 667 drugs by their human half-life from structure, scored by Spearman correlation on TDC’s scaffold split.

Classify drug-induced liver injury from a SMILES string

Drug-induced liver injury from structure

Classify whether a drug can cause liver injury, scored by AUROC on an FDA-derived set of 475 drugs.

Classify CYP3A4 inhibition from a SMILES string

CYP3A4 inhibition from structure

Classify whether a compound inhibits the CYP3A4 enzyme, scored by AUPRC over 12,328 compounds.

Predict Caco-2 cell effective permeability from a SMILES string

Caco-2 cell permeability from structure

Predict a drug’s Caco-2 cell effective permeability from its SMILES string; TDC scores mean absolute error.

Classify blood-brain barrier penetration from a SMILES string

Blood-brain barrier penetration from structure

Classify whether a compound crosses the blood-brain barrier, scored by AUROC on TDC’s scaffold split of 1,975 drugs.

Predict aqueous solubility (log mol/L) from a SMILES string

Aqueous solubility from structure

Predict aqueous solubility in log mol/L for 9,982 compounds from AqSolDB, scored by mean absolute error.

Classify Ames mutagenicity from a SMILES string

Ames mutagenicity from structure

Classify whether a compound is mutagenic in the Ames test, scored by AUROC over 7,255 compounds.

Predict rat acute oral toxicity LD50 from a SMILES string

Acute oral toxicity (LD50) from structure

Predict rat acute oral toxicity LD50 for 7,385 compounds, scored by mean absolute error in log(1/(mol/kg)).

Maximise the sitagliptin_mpo oracle within 10,000 calls

Sitagliptin MPO: same properties, different molecule

Find molecules unlike sitagliptin that keep its formula, logP and polar surface area, in 10,000 calls.

Maximise the mestranol_similarity oracle within 10,000 calls

Finding molecules similar to mestranol

Generate molecules with atom-pair similarity to mestranol above 0.75, within 10,000 oracle calls.

Maximise the jnk3 oracle within 10,000 calls

Designing JNK3 inhibitors under a 10,000-call budget

Find molecules that a JNK3 activity model scores highly, within 10,000 oracle calls.

Maximise the isomers_c9h10n2o2pf2cl oracle within 10,000 calls

Finding isomers of an awkward molecular formula

Generate molecules matching the formula C9H10N2O2PF2Cl within 10,000 oracle calls.

Maximise the gsk3b oracle within 10,000 calls

Designing GSK3β inhibitors under a 10,000-call budget

Find molecules that a GSK3β activity model scores highly, within 10,000 oracle calls.

Maximise the drd2 oracle within 10,000 calls

Designing molecules that score high on the DRD2 oracle

Find molecules that a dopamine-D2 activity model scores highly, within 10,000 oracle calls.

Estimate classically the operator Loschmidt echo of fixed 49–70-qubit circuits

Operator Loschmidt echo circuits on 49 to 70 qubits

Classically estimate an operator Loschmidt echo on 49- to 70-qubit circuits, where IBM hardware and classical simulations still disagree.

Score genes by how spatially variable their expression is

Detecting spatially variable genes

Rank genes by spatial variability in 50 simulated spatial transcriptomics datasets where the true variability is known.

Simulate realistic spatial transcriptomics data

Realistic simulators for spatial transcriptomics

Simulate spatial transcriptomics data that match ten real datasets on 23 distributional and downstream-analysis metrics.

Estimate per-spot cell-type proportions from a single-cell reference

Cell-type decomposition of spatial transcriptomics spots

Estimate the cell-type proportions in each spot of simulated spatial transcriptomics data, given a single-cell reference.

Predict one modality's profile from another in held-out cells

Predicting one single-cell modality from another

Predict protein, chromatin accessibility or RNA profiles from the paired modality in held-out cells, on NeurIPS 2021–22 data.

Predict differential expression for held-out compound–cell type pairs

Predicting drug responses across blood cell types

Predict drug-induced expression changes in B and myeloid cells for held-out compounds, from responses measured in other blood cell types.

Project cell-type labels from labelled batches onto a held-out batch

Cell-type label projection from a reference atlas

Train a cell-type classifier on labelled batches and annotate a held-out batch, across six CELLxGENE atlases.

Reduce scRNA-seq data to a 2D embedding for visualisation

Two-dimensional embeddings for single-cell visualisation

Embed single-cell expression data in two dimensions while preserving neighbourhoods, distances and densities.

Denoise scRNA-seq count matrices

Denoising single-cell RNA-seq counts

Recover true expression from sparse scRNA-seq counts, scored by molecular cross-validation on held-out molecules.

Prioritise ligand–target cell type interactions

Inferring cell-cell communication from single-cell data

Rank ligand–target cell type interactions in a breast cancer atlas against a ground truth inferred from cytokine activity.

Integrate scRNA-seq batches while conserving biological variation

Batch integration of single-cell atlases

Remove batch effects from six single-cell atlases while keeping cell-type variation, scored on 13 metrics against control methods.

Predict steel yield strength from composition

Predicting steel yield strength from composition

Predict the yield strength of a steel in MPa from its chemical formula alone, over 312 measured alloys.

Predict the highest optical phonon peak frequency

Predicting the highest phonon peak frequency of a crystal

Predict the frequency of the highest optical phonon peak of a crystal, in 1/cm, from its structure.

Predict perovskite cell formation energy from structure

Predicting perovskite formation energy from structure

Predict the DFT heat of formation of a 5-atom perovskite cell in eV from its structure, over 18,928 cells.

Predict PBE band gap from crystal structure

Predicting DFT band gaps from crystal structure

Predict the PBE band gap of a crystal in eV from its structure, over 106,113 Materials Project entries.

Predict formation energy from crystal structure

Predicting formation energy from crystal structure

Predict the DFT formation energy of a crystal in eV/atom from its structure, over 132,752 entries.

Predict log10 shear modulus from crystal structure

Predicting the shear modulus of a crystal

Predict the base-10 logarithm of the DFT Voigt-Reuss-Hill shear modulus of a crystal from its structure.

Predict exfoliation energy from crystal structure

Predicting exfoliation energy of 2D materials

Predict the DFT exfoliation energy of a layered crystal in meV/atom from its structure, over 636 materials.

Classify bulk metallic glass formation from composition

Predicting bulk metallic glass formation from composition

Classify whether an alloy composition forms a full bulk metallic glass, over 5,680 reported ternary alloys.

Predict experimental band gap from composition

Predicting experimental band gaps from a chemical formula

Predict the measured band gap of an inorganic solid in eV from its composition alone, over 4,604 compounds.

Crystal stability prediction on the WBM test set

Triaging hypothetical crystals for stability

Classify 215,488 hypothetical crystals as stable or not against a fixed DFT convex hull, scored by F1.

Predict refractive index from crystal structure

Predicting refractive index from crystal structure

Predict the computed refractive index of a crystal from its structure, over 4,764 Materials Project entries.

Find the lowest variational energy of the [4Fe-4S] (54e, 36o) Hamiltonian

Ground-state energy of the [4Fe-4S] iron–sulfur cluster

Find the lowest variational energy of a 54-electron, 36-orbital [4Fe-4S] cluster model; DMRG reaches about −327.24 Eh, the tracker's best entry −327.175.

Maximize the weakest focal intensity enhancement of an RGB metalens

Achromatic RGB metalens with 100 nm features

Design a 10 µm binary metalens that focuses 450, 550 and 650 nm light to one point, maximizing the weakest intensity enhancement.

Maximize the collected-power enhancement of an NV-center photon extractor

Photon extractor for a diamond nitrogen-vacancy center

Design a GaP nanostructure on diamond that maximizes the collected 637 nm light of a buried NV center, with features of at least 50 nm.

Maximize the worst-case relative efficiency of an 8-atom metagrating library

Broadband meta-atom library for visible metasurfaces

Design eight TiO₂ meta-atoms whose stitched grating keeps a high relative diffraction efficiency from 450 to 650 nm, with features of at least 60 nm.

Maximize efficiency × (1 − uniformity error) of a 7 × 7 beam splitter

Diffractive beam splitter into a 7 × 7 spot array

Design an etched silica splitter that divides a 632.8 nm beam into a 7 × 7 array of equal beams, maximizing efficiency times uniformity.

Maximize the worst-color efficiency of a Bayer color-sorting metasurface

Metasurface color sorter for a Bayer image sensor

Design a silicon-nitride metasurface that routes blue, green and red light into the pixels of a Bayer image sensor, with features of at least 80 nm.

Maximize the total face value of Galilean-moon flybys within four years

Global mapping of Jupiter's Galilean moons by flybys (GTOC6)

Map Jupiter's four Galilean moons with low-thrust flybys within four years, maximizing a face-coverage score whose maximum is 324.

Maximize the number of asteroid flybys before a final rendezvous

Asteroid billiard: most asteroid flybys before a rendezvous (GTOC4)

Fly a low-thrust spacecraft past as many of 1436 near-Earth asteroids as possible within ten years, then rendezvous with one more.

Maximize final mass per flight year for a four-asteroid rendezvous tour

Low-thrust rendezvous with four asteroid groups (GTOC2)

Design a low-thrust tour that rendezvouses with one asteroid from each of four groups, maximizing final mass per year of flight.

Maximize J = m_f |U_rel · v_ast| for a low-thrust kinetic impactor

Low-thrust kinetic impactor on asteroid 2001 TW229 (GTOC1)

Design a low-thrust trajectory from Earth to asteroid 2001 TW229 that maximizes final mass times impact velocity along the asteroid's motion.

Estimate classically Floquet Ising observables on 51- and 74-qubit heavy-hex

Floquet Ising dynamics on 51- and 74-qubit heavy-hex lattices

Classically estimate magnetization after up to 30 Floquet Ising cycles on 51 and 74 qubits, where simulations have not converged to the hardware data.

Design sequences that fold into the 100 Eterna100-V2 target structures

Eterna100-V2: designing RNA sequences for 100 target folds

Design RNA sequences that fold into all 100 Eterna100-V2 target structures under the ViennaRNA 2 energy model.

Maximize the normalized gradient scale length of a QI stellarator boundary

Simple-to-build quasi-isodynamic stellarator

Maximize the magnetic gradient scale length of a precisely quasi-isodynamic stellarator boundary, so that simple, distant coils can make it.

Pareto set of MHD-stable QI stellarators: aspect ratio vs gradient scale length

MHD-stable QI stellarators: compactness versus simple coils

Find a Pareto set of MHD-stable quasi-isodynamic stellarator boundaries that trades compactness against coil simplicity.

Minimize the maximum elongation of a stellarator boundary within shape limits

Least-elongated stellarator plasma boundary

Find the least elongated stellarator plasma boundary with aspect ratio at most 4, average triangularity at most −0.5 and edge iota per period at least 0.3.

Run upsample-FIR-downsample on long signals faster than scipy.signal.upfirdn

Faster upfirdn: upsample, filter and downsample long signals

Upsample, FIR-filter and downsample four signals of up to 5.3 million samples faster than scipy.signal.upfirdn, to a relative error of 1e-6.

Compute the thin SVD of a dense random matrix faster than numpy.linalg.svd

Faster thin singular value decomposition

Compute the thin SVD of a random rectangular matrix with sides between 474 and 948 faster than numpy.linalg.svd, with orthonormal factors.

Integrate Robertson's stiff kinetics to t = 1e7 faster than SciPy Radau

Faster stiff ODE solve: Robertson's chemical kinetics

Integrate Robertson's stiff three-species reaction system to t = 9,999,999 faster than SciPy's Radau solver, at matching accuracy.

Compute exp(A) for a 555 × 555 random matrix faster than scipy.linalg.expm

Faster matrix exponential

Compute the exponential of a 555 × 555 random matrix faster than scipy.linalg.expm, to a relative tolerance of 1e-5.

Fit Lasso (alpha = 0.1) on a 1990 × 3980 design faster than scikit-learn

Faster Lasso regression

Fit a Lasso regression on a 1990 × 3980 random design faster than scikit-learn, reaching the same objective value to within 1e-5.

Solve a 23-state, 46-step Kalman smoothing QP faster than CVXPY

Faster Kalman smoothing posed as a quadratic program

Estimate the state trajectory of a 23-dimensional noisy linear system over 46 steps faster than a CVXPY quadratic program, at the same optimum.

Integrate the stiff HIRES kinetics model to t = 370 faster than SciPy Radau

Faster stiff ODE solve: the HIRES photochemistry model

Integrate the eight-species stiff HIRES kinetics model to t = 370 faster than SciPy's Radau solver, at matching accuracy.

Solve a dense convex QP with 278 variables faster than CVXPY with OSQP

Faster convex quadratic programming

Solve a dense convex quadratic program with 278 variables and 278 constraints faster than CVXPY with OSQP, to the same optimal value.

Solve a 240-point semi-discrete Burgers' equation to t = 5 faster than RK45

Faster method-of-lines solve of Burgers' equation

Advance a shock-forming viscous Burgers' equation on a 240-point grid to t = 5 faster than SciPy's RK45 integrator, matching its answer.

Compute all-pairs shortest paths on a 352-node graph faster than SciPy

Faster all-pairs shortest paths on a weighted graph

Compute all-pairs shortest-path distances of a 352-node undirected weighted graph faster than SciPy's Dijkstra-based shortest_path.