Caco-2 cell permeability from structure

The Caco-2 monolayer is the standard laboratory stand-in for the wall of the human gut: how fast a compound crosses it approximates how fast the compound will be absorbed. This benchmark asks for that rate straight from the molecular structure, with no assay.

Task

Given a drug as a SMILES string, predict its Caco-2 cell effective permeability. The labels are logarithmic — the benchmark files run from $-7.76$ to $-3.51$, consistent with $\log_{10}$ of the apparent permeability $P_\mathrm{app}$ in cm/s — and TDC scores a submission by mean absolute error, $\mathrm{MAE} = \frac{1}{n}\sum_i |y_i - \hat{y}_i|$, so lower is better.

Background

Caco-2 permeability is measured early in a discovery programme because poor absorption is one of the common reasons an otherwise promising compound fails. The dataset TDC uses was assembled by Wang et al. (2016), who collected published Caco-2 measurements and fitted their own model to them; TDC wraps those 906 compounds as the first benchmark of its ADMET group. Being small and regression-shaped, it is a frequent first test for molecular representations: fingerprints, descriptors and pretrained graph encoders are all routinely compared on it.

Data and evaluation

TDC ships the group as one archive (admet_group, Harvard Dataverse file 4426004), loaded with admet_group(path='data/') from the tdc.benchmark_group module. The group’s caco2_wang folder holds 728 train_val and 182 test rows (910 rows, 906 distinct SMILES; TDC’s page lists 906 drugs).

The split is a scaffold split: TDC holds out a fixed 20% test set (here 182 of 910 rows in the group’s files), and the remaining train_val part is split again per seed by group.get_train_valid_split(benchmark=name, split_type='default', seed=seed) into 87.5% training and 12.5% validation. The leaderboard asks for five independent runs with seeds 1, 2, 3, 4 and 5; group.evaluate_many then reports the mean and the standard deviation of the per-seed test scores, rounded to three decimals. The test set never changes, so only the training split moves with the seed.

The metric is mean absolute error, $\mathrm{MAE} = \frac{1}{n}\sum_{i=1}^{n} |y_i - \hat{y}_i|$, computed by TDC’s Evaluator(name='mae') on the test set; lower is better.

Official evaluation code: tdc/benchmark_group/base_group.py in the TDC repository (MIT licence), whose evaluate and evaluate_many methods are what the leaderboard numbers come from. The dataset page gives the licence as "Not Specified. CC BY 4.0".

What is known

• Best on the leaderboard: CaliciBoost, MAE $0.256 \pm 0.006$, a gradient-boosting model over several molecular representations (Le, Ren et al., 2025).
• Next: XG Boost, $0.274 \pm 0.004$ (submitted by Rob Learsch), then MapLight, $0.276 \pm 0.005$, CatBoost over ECFP, Avalon and ErG fingerprints with physicochemical descriptors (Notwell and Wood, 2023).
• Twenty-four entries are listed. Classical fingerprint-plus-boosting models hold the top places; the pretrained graph models that lead several other endpoints in this group do not lead here (MiniMol is fifteenth at $0.350 \pm 0.018$).

What would count as progress

• A mean MAE on TDC’s fixed test set that beats CaliciBoost (MAE 0.256), averaged over the official seeds 1-5 and produced by group.evaluate_many, with the five per-seed numbers reported as well.
• The training and inference code, the environment and the fitted model, so that the five runs can be repeated end to end from the train_val part alone.
• Evidence that the gain is larger than run-to-run noise: the gap between the top two entries here is about 0.02 MAE, so a difference smaller than the reported standard deviation is not a result.
• An error analysis that says which chemistry the model gets wrong, since 182 test compounds leave room for a lucky split.

References

1. Wang, N.-N. et al., "ADME Properties Evaluation in Drug Discovery: Prediction of Caco-2 Cell Permeability Using a Combination of NSGA-II and Boosting", Journal of Chemical Information and Modeling 56 (2016), 763–773. doi:10.1021/acs.jcim.5b00642
2. Le, H. V., Ren, W. et al., "CaliciBoost: Performance-Driven Evaluation of Molecular Representations for Caco-2 Permeability Prediction", arXiv:2506.08059 (2025). arxiv.org/abs/2506.08059
3. Notwell, J. H. and Wood, M. W., "ADMET property prediction through combinations of molecular fingerprints", arXiv:2310.00174 (2023). arxiv.org/abs/2310.00174
4. Huang, K. et al., "Artificial intelligence foundation for therapeutic science", Nature Chemical Biology 18 (2022), 1033–1036. doi:10.1038/s41589-022-01131-2

Source

Imported from TDC ADMET benchmark group, Caco2_Wang, curated by Therapeutics Data Commons (Harvard). Open according to the source as of 2026-10-20. The statement above is written in our own words; see the source for the original wording, history and discussion.