Drug-induced liver injury from structure

Classify whether a drug can cause liver injury, scored by AUROC on an FDA-derived set of 475 drugs.

shoal State the problem, from Therapeutics Data Commons ADMET benchmark group 814b3b0 · 2周前历史

56 行 · 4.9 KB 原始文件永久链接历史

1	# Drug-induced liver injury from structure
2
3	Liver injury has been the single most common reason for withdrawing an approved drug from the market, and it is notoriously hard to see coming. This benchmark asks whether a drug can cause it, from structure alone, on a small set curated from FDA records.
4
5	## Task
6
7	Given a drug as a SMILES string, predict whether it can cause liver injury ($1$) or not ($0$). Submissions are scored by AUROC on TDC’s held-out test set, so higher is better.
8
9	## Background
10
11	The set was aggregated from the US FDA’s National Center for Toxicological Research and used by Xu et al. (2015) for a deep-learning model of drug-induced liver injury; TDC includes its 475 drugs, the smallest benchmark in the ADMET group. Liver toxicity is often idiosyncratic — dependent on dose, metabolism and the patient rather than on structure alone — so high scores here deserve scepticism, and the endpoint is as much a test of honest evaluation as of chemistry.
12
13	## Data and evaluation
14
15	TDC ships the group as one archive (`admet_group`, Harvard Dataverse file 4426004), loaded with
16	`admet_group(path='data/')` from the `tdc.benchmark_group` module. The group’s `dili` folder holds 379 `train_val` and 96 `test` rows (475 distinct SMILES, matching TDC’s page, with no duplicates); 49.7% of labels are positive, and 50 of the 96 test drugs are.
17
18	The split is a scaffold split: TDC holds out a fixed 20% test set (here 96 of 475 rows in
19	the group’s files), and the remaining `train_val` part is split again per seed by
20	`group.get_train_valid_split(benchmark=name, split_type='default', seed=seed)` into 87.5% training
21	and 12.5% validation. The leaderboard asks for five independent runs with seeds 1, 2, 3, 4 and 5;
22	`group.evaluate_many` then reports the mean and the standard deviation of the per-seed test scores,
23	rounded to three decimals. The test set never changes, so only the training split moves with the seed.
24
25	The metric is AUROC, the area under the receiver operating characteristic curve, computed by TDC’s `Evaluator(name='roc-auc')`; it equals the probability that a positive compound is scored above a negative one, and higher is better. Submissions pass continuous scores, not hard labels.
26
27	Official evaluation code: [`tdc/benchmark_group/base_group.py`](https://github.com/mims-harvard/TDC/blob/main/tdc/benchmark_group/base_group.py)
28	in the TDC repository (MIT licence), whose `evaluate` and `evaluate_many` methods are what the
29	leaderboard numbers come from. The dataset page gives the licence as "Not Specified. CC BY 4.0".
30
31	## What is known
32
33	- Best on the leaderboard: MiniMol, AUROC $0.956 \pm 0.006$ (Kläser, Banaszewski et al., 2024).
34	- Next: ZairaChem, $0.925 \pm 0.005$ (Turon et al., 2023), then AttrMasking, $0.919 \pm 0.008$ (Hu et al., 2020).
35	- Twenty entries are listed, from AUROC $0.956$ to $0.700$. The top score stands out from the pack, but on 96 test drugs one AUROC point is worth about half a compound, so the ranking below first place is close to arbitrary.
36
37	## What would count as progress
38
39	- A mean AUROC on TDC’s fixed test set that beats MiniMol (AUROC 0.956), averaged over the official
40	seeds 1-5 and produced by `group.evaluate_many`, with the five per-seed numbers reported as well.
41	- The training and inference code, the environment and the fitted model, so that the five runs can be
42	repeated end to end from the `train_val` part alone.
43	- Evidence that the gain is larger than run-to-run noise: the gap between the top two entries here is
44	0.031 AUROC, on a test set of 96 drugs, so a difference smaller than the reported standard deviation is not a result.
45	- Per-seed scores, and ideally a check on drugs withdrawn for hepatotoxicity that are not in the dataset, since 96 test compounds cannot settle this endpoint on their own.
46
47	## References
48
49	1. Xu, Y. et al., "Deep Learning for Drug-Induced Liver Injury", Journal of Chemical Information and Modeling 55 (2015), 2085–2093. [doi:10.1021/acs.jcim.5b00238](https://doi.org/10.1021/acs.jcim.5b00238)
50	2. Kläser, K., Banaszewski, B. et al., "MiniMol: A Parameter-Efficient Foundation Model for Molecular Learning", arXiv:2404.14986 (2024). [arxiv.org/abs/2404.14986](https://arxiv.org/abs/2404.14986)
51	3. Turon, G. et al., "First fully-automated AI/ML virtual screening cascade implemented at a drug discovery centre in Africa", Nature Communications 14 (2023), 5736. [doi:10.1038/s41467-023-41512-2](https://doi.org/10.1038/s41467-023-41512-2)
52	4. Huang, K. et al., "Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development", NeurIPS Datasets and Benchmarks (2021), arXiv:2102.09548. [arxiv.org/abs/2102.09548](https://arxiv.org/abs/2102.09548)
53
54	## Source
55
56	Imported from [TDC ADMET benchmark group, DILI](https://tdcommons.ai/benchmark/admet_group/22dili/), curated by Therapeutics Data Commons (Harvard). Open according to the source as of 2026-10-20. The statement above is written in our own words; see the source for the original wording, history and discussion.