Cell-type label projection from a reference atlas

New single-cell datasets are usually annotated by transferring cell-type labels from an existing, already labelled reference. This benchmark measures how accurately a method projects labels onto a batch it has never seen, across six large atlases.

Task

Each dataset contains two or more batches of single-cell RNA-seq profiles with matching, manually curated cell-type labels. One batch is held out as the test set. A method receives the training cells (expression and labels) and the test cells (expression only), and must predict a cell-type label for every test cell. Predictions are compared with the held-out true labels.

Background

Annotating by clustering and marker genes is hard to reconcile across atlas-scale datasets, because annotations made by different groups may not match. Reference mapping instead trains a classifier, or a shared latent model, on an annotated reference and projects its labels onto the query. Methods range from logistic regression on PCA coordinates to deep generative models such as scANVI and transfer-learning approaches such as scArches. The task was set up by Nikolay Markov, Scott Gigante and Robrecht Cannoodt within Open Problems.

Data and evaluation

• Data: six CELLxGENE Census collections (diabetic kidney disease, GTEx v9, HypoMap, Immune Cell Atlas, mouse pancreatic islet atlas, Tabula Sapiens). Processed train, test and solution files are in the public bucket s3://openproblems-data/resources/task_label_projection/datasets/. The task code is MIT-licensed; dataset licences are not restated on the task page.
• Split: for each dataset one batch is held out as the test set (num_test_batches = 1). The batch is drawn at random by the processing script, and the processing workflow sets no seed, so the split is defined by the released files rather than by a seed.
• Metric: four metrics in release v2.0.0: accuracy and the macro, micro and weighted F1 scores. For each dataset and metric the raw value $x$ is rescaled between the worst and best control method (random labels, majority vote, true labels), $s=\min\{1,\max\{0,(x-c_{\text{worst}})/(c_{\text{best}}-c_{\text{worst}})\}\}$, and the overall score is the mean of $s$ over all 6 × 4 pairs, with failed runs counted as 0. The leaderboard shows it to two decimals.
• Code: openproblems-bio/task_label_projection.

What is known

On the v2.0.0 leaderboard (as of 2026-10-20):

• scANVI with scArches reference mapping leads at 0.79 (0.7931 before rounding, recomputed from the published results).
• Seurat TransferData, logistic regression and plain scANVI follow at 0.78, a multilayer perceptron at 0.77 and XGBoost at 0.76: simple classifiers are close to the best.
• Foundation-model approaches trail: SCimilarity at 0.45 (best parameter set), fine-tuned scGPT at 0.34 and UCE at 0.02.
• The majority-vote control scores 0.05, so the gap between a trivial and a good method is large, but the top is crowded.

What would count as progress

• An overall score above 0.7931 on the v2.0.0 datasets and metrics, computed with the task's run_benchmark workflow, with the method contributed as a component so that it can be rerun.
• A gain driven by macro F1, that is, better accuracy on rare cell types, rather than by the already high accuracy on common ones.
• A gain that holds on each of the six datasets separately, not only on average.

References

1. Lotfollahi M, Naghipourfar M, Luecken MD, et al. Mapping single-cell data to reference atlases by transfer learning. Nature Biotechnology 40, 121–130 (2022). doi:10.1038/s41587-021-01001-7
2. Xu C, Lopez R, Mehlman E, et al. Probabilistic harmonization and annotation of single-cell transcriptomics data with deep generative models. Molecular Systems Biology 17, e9620 (2021). doi:10.15252/msb.20209620
3. Hao Y, Hao S, Andersen-Nissen E, et al. Integrated analysis of multimodal single-cell data. Cell 184, 3573–3587 (2021). doi:10.1016/j.cell.2021.04.048
4. Luecken MD, Gigante S, Burkhardt DB, Cannoodt R, et al. Defining and benchmarking open problems in single-cell analysis. Nature Biotechnology 43, 1035–1040 (2025). doi:10.1038/s41587-025-02694-w

Source

Imported from Open Problems in Single-Cell Analysis: Label Projection (task_label_projection), curated by the Open Problems in Single-Cell Analysis consortium. Open according to the source as of 2026-10-20 (leaderboard release v2.0.0). The statement above is written in our own words; see the source for the original wording, history and discussion.