Predicting one single-cell modality from another

Some assays measure two molecular layers in the same cell, such as RNA together with surface proteins or chromatin accessibility. This benchmark asks how well one layer can be predicted from the other in cells a model has not seen, a test of how much of the regulatory link between the layers a model has captured.

Task

Each dataset pairs gene expression (GEX) with a second modality measured in the same cells: surface protein abundance (ADT, from CITE-seq) or chromatin accessibility (ATAC, from 10x Multiome). Each pair is used in both directions, for example GEX → ADT and ADT → GEX. A method receives both modalities for the training cells and only the input modality for the test cells, and must predict the full profile of the other modality for every test cell.

Background

The task was one of three in the NeurIPS 2021 multimodal single-cell integration competition, built on a bone-marrow dataset generated for the purpose (Luecken et al., 2021; Lance et al., 2022). Open Problems keeps it as a living benchmark and has added the 2022 Kaggle multimodal competition data and further methods, including published models such as BABEL, sciPENN and scButterfly and the first-place Kaggle solution.

Data and evaluation

• Data: eight datasets: the NeurIPS 2021 bone-marrow CITE-seq and Multiome data (GEO GSE194122) and the NeurIPS 2022 Kaggle CITE-seq and Multiome data, each in both directions. ATAC is reduced to the 10,000 most variable peaks. Files are in s3://openproblems-data/resources/task_predict_modality/datasets/. The task code is MIT-licensed; dataset licences are not restated on the task page.
• Split: the competitions' predefined train/test labels are used where present (otherwise a quarter of the batches is held out), and the test set is subsampled to 1,000 cells, weighted towards rare cell types. The processing step uses seed 1.
• Metric: eight metrics in release v0.2.0: RMSE, MAE, Pearson and Spearman correlation per cell, per gene and over the flattened matrix. Each is rescaled per dataset between the worst and best control method, $s=\min\{1,\max\{0,(x-c_{\text{worst}})/(c_{\text{best}}-c_{\text{worst}})\}\}$, and the overall score is the mean of $s$ over all 8 × 8 pairs, with failed runs counted as 0. The leaderboard shows it to two decimals.
• Code: openproblems-bio/task_predict_modality.

What is known

On the v0.2.0 leaderboard (as of 2026-10-20):

• k-nearest-neighbour regression in Python leads at 0.37 (0.3666 before rounding, recomputed from the published results).
• CellMapper variants follow at 0.36, then "Novel", an encoder–decoder model from the 2021 competition, at 0.35, and the first-place solution of the 2022 Kaggle challenge at 0.34.
• Several published deep models score lower (sciPENN 0.13, scButterfly 0.12, BABEL 0.07), below the control that predicts each feature's mean (0.22).

That a plain nearest-neighbour regression is the best method suggests that much of the cross-modal signal is still unexploited.

What would count as progress

• An overall score above 0.3666 on the v0.2.0 datasets and metrics, computed with the task's run_benchmark workflow, with code that reproduces it.
• A method that beats k-nearest-neighbour regression on each of the eight datasets, that is, in both directions of every modality pair.
• A gain confirmed on the per-gene correlation metrics, which reward capturing how each feature varies across cells rather than its average level.

References

1. Luecken MD, Burkhardt DB, Cannoodt R, et al. A sandbox for prediction and integration of DNA, RNA, and proteins in single cells. NeurIPS Datasets and Benchmarks Track (2021). proceedings
2. Lance C, Luecken MD, Burkhardt DB, et al. Multimodal single cell data integration challenge: results and lessons learned. Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, PMLR 176, 162–176 (2022). proceedings.mlr.press/v176/lance22a
3. Luecken MD, Gigante S, Burkhardt DB, Cannoodt R, et al. Defining and benchmarking open problems in single-cell analysis. Nature Biotechnology 43, 1035–1040 (2025). doi:10.1038/s41587-025-02694-w

Source

Imported from Open Problems in Single-Cell Analysis: Predict Modality (task_predict_modality), curated by the Open Problems in Single-Cell Analysis consortium. Open according to the source as of 2026-10-20 (leaderboard release v0.2.0). The statement above is written in our own words; see the source for the original wording, history and discussion.