Ames mutagenicity from structure

Classify whether a compound is mutagenic in the Ames test, scored by AUROC over 7,255 compounds.

shoal State the problem, from Therapeutics Data Commons ADMET benchmark group 21880a4 · 2周前历史

55 行 · 4.6 KB 原始文件永久链接历史

1	# Ames mutagenicity from structure
2
3	The Ames test exposes bacteria to a compound and looks for mutations; a positive result usually ends a drug programme. This benchmark asks for the outcome of that assay from the structure alone, over the largest public collection of Ames results.
4
5	## Task
6
7	Given a compound as a SMILES string, predict whether it is mutagenic ($1$) or not ($0$) in the Ames assay. Submissions are scored by AUROC, so higher is better.
8
9	## Background
10
11	The Ames bacterial reverse mutation assay is the standard first screen for genetic toxicity, required by regulators before a compound reaches people. Xu et al. (2012) aggregated results from four published collections into the set TDC uses, 7,255 compounds, which is large and almost balanced (54.6% mutagenic). Because mutagenicity is often driven by specific reactive substructures, this endpoint is a good test of whether a model has learned chemistry rather than a global average.
12
13	## Data and evaluation
14
15	TDC ships the group as one archive (`admet_group`, Harvard Dataverse file 4426004), loaded with
16	`admet_group(path='data/')` from the `tdc.benchmark_group` module. The group’s `ames` folder holds 5,821 `train_val` and 1,457 `test` rows (7,278 rows, 7,255 distinct SMILES, matching TDC’s page); 870 of the 1,457 test compounds are mutagenic.
17
18	The split is a scaffold split: TDC holds out a fixed 20% test set (here 1,457 of 7,278 rows in
19	the group’s files), and the remaining `train_val` part is split again per seed by
20	`group.get_train_valid_split(benchmark=name, split_type='default', seed=seed)` into 87.5% training
21	and 12.5% validation. The leaderboard asks for five independent runs with seeds 1, 2, 3, 4 and 5;
22	`group.evaluate_many` then reports the mean and the standard deviation of the per-seed test scores,
23	rounded to three decimals. The test set never changes, so only the training split moves with the seed.
24
25	The metric is AUROC, the area under the receiver operating characteristic curve, computed by TDC’s `Evaluator(name='roc-auc')`; it equals the probability that a positive compound is scored above a negative one, and higher is better. Submissions pass continuous scores, not hard labels.
26
27	Official evaluation code: [`tdc/benchmark_group/base_group.py`](https://github.com/mims-harvard/TDC/blob/main/tdc/benchmark_group/base_group.py)
28	in the TDC repository (MIT licence), whose `evaluate` and `evaluate_many` methods are what the
29	leaderboard numbers come from. The dataset page gives the licence as "Not Specified. CC BY 4.0".
30
31	## What is known
32
33	- Best on the leaderboard: ZairaChem, AUROC $0.871 \pm 0.002$, an automated QSAR pipeline (Turon et al., 2023).
34	- Next: ADMETrix, $0.870 \pm 0.006$, then MapLight + GNN, $0.869 \pm 0.002$ — the top three are separated by $0.002$ AUROC.
35	- Twenty entries are listed, from AUROC $0.871$ to $0.716$. The leaderboard has been flat near $0.87$ across quite different methods, which suggests the remaining error is in the data as much as in the models.
36
37	## What would count as progress
38
39	- A mean AUROC on TDC’s fixed test set that beats ZairaChem (AUROC 0.871), averaged over the official
40	seeds 1-5 and produced by `group.evaluate_many`, with the five per-seed numbers reported as well.
41	- The training and inference code, the environment and the fitted model, so that the five runs can be
42	repeated end to end from the `train_val` part alone.
43	- Evidence that the gain is larger than run-to-run noise: the gap between the top two entries here is
44	0.001 AUROC, so a difference smaller than the reported standard deviation is not a result.
45	- A check that the gain holds on compounds whose scaffolds are absent from the training split, since structural alerts transfer easily and can flatter a model.
46
47	## References
48
49	1. Xu, C. et al., "In silico Prediction of Chemical Ames Mutagenicity", Journal of Chemical Information and Modeling 52 (2012), 2840–2847. [doi:10.1021/ci300400a](https://doi.org/10.1021/ci300400a)
50	2. Turon, G. et al., "First fully-automated AI/ML virtual screening cascade implemented at a drug discovery centre in Africa", Nature Communications 14 (2023), 5736. [doi:10.1038/s41467-023-41512-2](https://doi.org/10.1038/s41467-023-41512-2)
51	3. Notwell, J. H. and Wood, M. W., "ADMET property prediction through combinations of molecular fingerprints", arXiv:2310.00174 (2023). [arxiv.org/abs/2310.00174](https://arxiv.org/abs/2310.00174)
52
53	## Source
54
55	Imported from [TDC ADMET benchmark group, AMES](https://tdcommons.ai/benchmark/admet_group/21ames/), curated by Therapeutics Data Commons (Harvard). Open according to the source as of 2026-10-20. The statement above is written in our own words; see the source for the original wording, history and discussion.