Designing JNK3 inhibitors under a 10,000-call budget

c-Jun N-terminal kinase 3 is a neuronal kinase and a long-standing drug target. This task asks a design algorithm to find molecules that a random forest trained on measured JNK3 activities scores highly, while calling that model at most 10,000 times.

Task

Propose molecules as SMILES strings and maximise the JNK3 oracle, a random forest over ECFP6 fingerprints fitted to ExCAPE-DB activity data, returning a score in $[0, 1]$. The run is scored by AUC top-10 over a 10,000-call budget.

Background

JNK3 is the harder sibling of the GSK3β oracle: its active set is smaller and the random forest gives almost no gradient to follow, so most methods plateau early. In the PMO study it is one of the oracles that separates the field most sharply, with the leader more than 0.1 ahead of the second place, which makes it a good test of whether a method can explore rather than exploit.

Data and evaluation

• Data: there is no training set to fit. Any database a method needs is restricted to ZINC 250k, which ships with the repository, so no method gets an advantage from a richer corpus.
• Budget and seeds: 10,000 oracle calls per run, and five independent runs with different random seeds (python run.py METHOD --task production --n_runs 5 --oracles jnk3). Report the mean and standard deviation over the runs. The budget counts distinct valid canonical SMILES: a molecule queried twice is served from the cache for free, and an invalid SMILES scores 0.
• Metric: AUC top-10. The mean score of the ten best molecules found so far is recorded every 100 calls, the curve against the number of calls is integrated by the trapezoid rule and divided by the 10,000-call budget, so the value lands in $[0, 1]$ and higher is better. A run that stops early keeps its final top-10 value for the rest of the budget. Unlike a plain top-10 average, the metric rewards reaching a good score early.
• Official evaluation code: https://github.com/wenhao-gao/mol_opt (MIT), where main/optimizer.py holds the budget accounting and the AUC. Oracles come from the Therapeutics Data Commons via PyTDC (MIT); the ZINC database is free to use.

What is known

• Best in the PMO paper's table: REINVENT at 0.783 ± 0.023, well clear of LSTM HC at 0.661 ± 0.039 and REINVENT on SELFIES at 0.631 ± 0.064.
• Later work on the same protocol reports higher: Mol GA at 0.835 ± 0.040 and Genetic GFN at 0.764 ± 0.069 (five runs, 10,000 calls).
• A random screen of ZINC 250k scores 0.238, the largest relative gap to the leader among the bioactivity oracles.
• Variance is the dominant problem: reported standard deviations reach 0.14, so single-run results on this oracle are not evidence.

What would count as progress

• A mean AUC top-10 above 0.783 on jnk3 from five runs under the official protocol, reported with the per-run values and the standard deviation. To beat the field rather than the 2022 table, the number to pass is 0.835 (Mol GA).
• The run logs, which record every scored molecule in order, so the AUC and the oracle-call count can be recomputed rather than trusted.
• Code in the repository's method layout, so the result can be re-run on the other 22 oracles: a method that wins on one oracle and loses elsewhere is the pattern PMO was built to expose.

References

1. W. Gao, T. Fu, J. Sun and C. W. Coley, "Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization", NeurIPS 2022 Datasets and Benchmarks (2022). https://arxiv.org/abs/2206.12411
2. Y. Li, L. Zhang and Z. Liu, "Multi-objective de novo drug design with conditional graph generative model", Journal of Cheminformatics 10, 33 (2018). https://doi.org/10.1186/s13321-018-0287-6
3. M. Olivecrona, T. Blaschke, O. Engkvist and H. Chen, "Molecular de-novo design through deep reinforcement learning", Journal of Cheminformatics 9, 48 (2017). https://doi.org/10.1186/s13321-017-0235-x
4. A. Tripp and J. M. Hernández-Lobato, "Genetic algorithms are strong baselines for molecule generation" (2023). https://arxiv.org/abs/2310.09267
5. H. Kim, M. Kim, S. Choi and J. Park, "Genetic-guided GFlowNets for Sample Efficient Molecular Optimization", NeurIPS 2024 (2024). https://arxiv.org/abs/2402.05961

Source

Imported from PMO (mol_opt), the jnk3 oracle, curated by Wenhao Gao, Tianfan Fu, Jimeng Sun and Connor W. Coley. Open according to the source as of 2026-10-20. The statement above is written in our own words; see the source for the original wording, history and discussion.