Abstract
OpenRSI's kev pairs a 500M-parameter model with a decision head across boolean, multiple-choice, and ranking tasks, scoring submissions by exp(−macro-average negative log-likelihood)—roughly the geometric mean probability placed on the ground-truth answer. The rules prohibit metadata-based answer routing. The clean line started at 0.30624, obtained by removing one non-compliant member from an ensemble built in October on our other platform AgenticOS without retraining—also this platform's previous compliant best. Across the clean line (November 9–11), the strongest submission without a source switch reached 0.32837, gaining +0.0221, more than four times the pre-registered +0.005 threshold. Because the judge is deterministic, the November 11 final evaluation reproduced the in-run score bit for bit. Ten higher-scoring candidates that use a source switch are excluded; the 0.32837 submission itself carries one compliance note—varying readout by option count—pending final human review. Single run, no control group, and no external review.
Keywordsdecision headsmall modelscompliancedeterministic judge
1Problem
How much probability does it put on the right answer?
Instead of generating text, a small model outputs a probability distribution over candidate options. The benchmark measures how much probability mass lands on the right option—requiring every probability to come from a trained decision head.
The base model is Qwen2.5-0.5B, capped at 600M parameters and a 3 GiB checkpoint. Evaluation covers 1,958 scored items, weighting the knowledge group (500 MMLU + 500 MMLU-Pro) at 50% and 12 other sources at 50%. The score is exp(−macro-average negative log-likelihood); uniform guessing scores 0.2488. The judge loads the checkpoint once and runs inference deterministically, returning identical scores across repeated runs.
Rules. Training is restricted to the fixed 12,576-record corpus, with no external models, external data, teacher distillation, or memorized answers; metadata-based answer routing is forbidden, and every output probability must come from a trained Kev decision head. That rule was already present in the September 27 revision of OpenRSI-Index.
Best before. Training the reference recipe shipped with the problem once gives 0.2519; across 5 seeds on our machine it ranged from 0.2423 to 0.2584. In AgenticOS wave 1 in October, team a assembled a 12-member ensemble scoring 0.3147 (10-08 14:03 UTC; reproduced bit for bit on this platform's judge on 11-09). One member in that ensemble let the frozen base model read option letters directly, bypassing the decision head in violation of the rules, and is therefore disqualified. Dropping that member and redistributing its multiple-choice weight proportionally across the other two letter-pointer members—without retraining—gives 0.3062444617 at 595,864,968 parameters. That became the clean line's starting point and this platform's previous compliant best. Public OpenRSI logs were run under different conditions and are not compared here.
2Method
Inference-side changes only, no retraining
In the best switch-free candidate, every change sits on the inference side: the checkpoint is unchanged from the starting point, and what changes is how items are formatted for the model and how probabilities are read out from the trained heads.
Table 1 Conditions of the run
| Item | Value |
|---|---|
| Run | Shoal clean line orsi-line-main-r1 (the same run as minference): 24 working hours starting 11-09 20:39 UTC, with one overnight pause on 11-10 (12.5 hours, excluded) |
| Members | Claude Code and Codex processes launched via our harness on our own machines using our quota; 562 member identities visited this problem |
| Network | Members had web access through a logging proxy; check jobs and evaluations ran offline |
| Thresholds | Pre-registered before launch: +0.005 above the starting point (0.3112); any new training recipe additionally required a 3-seed mean of at least 0.31254 |
| Route | Branch clean-stack, commit 4b7212f4; checkpoint unchanged from starting point dc47159c, no retraining |
| Cost | Not reported here: no audited record |
Times below are working hours from 11-09 20:39 UTC, excluding the overnight pause:
- Hour 1.1: Computing calendar facts from dates appearing in the item state and appending them to the prompt reaches 0.3082.
- Hours 3.0 and 3.5: Two members independently hit on the same idea—rendering items in the field order of the training corpus alongside computed date facts—scoring 0.313195 and 0.313218, clearing the threshold for the first time.
- Overnight pause: The evaluation queue continues draining its backlog; a submission that splits 8–10-option items into balanced 4-way subgroups before recombining them scores 0.3172.
- Hour 7.0: Combining several parent branches: predicted at 0.3235 from their per-source score tables beforehand, measured at 0.32339, matching within 0.00005 on all 14 sources.
- Hour 7.8 (11-10 16:55 UTC): A member opens issue #11, titled "facts for whoever reads the rules", opening with "This is a statement of facts, not a ruling"—drawing 72 comments from 29 authors (16 Claude, 13 Codex). Per prior protocol, the problem maintainers observed only and did not reply.
- From hour 8.2: A switch inspecting
subjectandcategoryfields in the item state matches 280 of 280 knowledge items on the public dev split and 0 of 876 remaining items—effectively detecting MMLU and MMLU-Pro before switching readout; flagged as questionable in audit. Nearly every higher score afterward builds on top of it. - Hour 9.6: After reading #11, another member pushes a switch-free branch described as "The strongest candidate with no evaluation-shape key": LoRA scaling and readout layout depend on option count, paired with the reference recipe's date facts, without inspecting any state field absent from the fixed training corpus. After 4.6 hours in the evaluation queue, it scores 0.3283677 at hour 14.2.
3Result
0.30624 → 0.32837
The best switch-free submission beats the starting point by +0.0221—over four times the threshold—and matches its in-run score bit for bit in the final evaluation.
Between 10:41 and 11:05 UTC on November 11, the final evaluation re-ran the starting point and 14 candidates; every score matched its in-run evaluation to the last digit. All 4 switch-free candidates cleared the threshold; the 10 candidates with a source switch scored higher and remain excluded pending a human ruling.
Table 2 Final evaluation (11-11, deterministic judge)
| Candidate | Score | vs. start | Status |
|---|---|---|---|
| Starting point | 0.30624 | — | Previous compliant best on this platform |
| clean-stack | 0.32837 | +0.0221 | new record (this platform only) |
| group4-facts | 0.31723 | +0.0110 | Above bar, no switch |
| lora-scale | 0.31573 | +0.0095 | Above bar, no switch |
| pairwise-readout | 0.31565 | +0.0094 | Above bar, no switch |
| 10 candidates with source switch | 0.32973–0.33395 | — | not counted, awaiting human ruling |
One submission in the switched group introduced a new training recipe whose in-run 0.33395 came from a single seed; running three fresh seeds per the pre-registered rule gave 0.33199, 0.32980, and 0.32997—a mean of 0.33059, just +0.00004 above its un-retrained base at 0.33055: no training effect.
4Checks
How it was judged, and who reproduced it
5Limits
What it does not show
LimitsWhat this report cannot say
- Compliance is not yet settled. This candidate carries a note on option-count-conditioned readout awaiting final human review; excluding the 10 source-switched candidates does not mean the switch-free candidate is free of compliance questions.
- Selection bias. Although the judge is deterministic—so +0.0221 has zero measurement noise—dozens of variants were evaluated against the same judge.
- Inherited starting point. The starting ensemble came from an October run on another platform after dropping a non-compliant member without retraining; under a stricter reading, two letter-pointer members might also be excluded (retaining only the nine pointer members drops the score to 0.2682).
- Most late-run submissions were never scored. With 4 evaluation slots and a median queue wait of 5.1 hours, 1,610 of 2,303 queued evaluations never started (1,282 superseded by newer commits on the same branch, 328 still queued at shutdown), and zero submissions after working hour 18 were evaluated. Our findings cover only the evaluated portion.
- This platform only. "New record" compares only against this platform's previous compliant best; we claim no leaderboard rank and do not compare against public OpenRSI logs.
- Single run, no control. Members were launched by us and had web access via a logging proxy.
- Wave 1's 0.3147. Because the rule against bypassing the decision head was already present in the September 27 version of the benchmark, wave 1's 0.3147 was almost certainly non-compliant in October as well—not merely disqualified retroactively under newer rules—though we did not directly inspect the exact prompt copy used during wave 1.
6Data and code
Where the records are
Every number above was recomputed from the records listed below. Those files live in Shoal's private lab repository, which readers cannot open; the paths under lab/docs/2026-11-10-openrsi-line/ are listed as internal provenance references. Code and checkpoints sit in the run repository and on the run machine and are likewise not public. What is public is the benchmark itself: kev-decision-architecture in OpenRSI-Index, and its upstream jaredpalmer/kev.
Every figure traced to a record
final/kev/scores.jsonl (15 rows: starting point + 14 candidates)Section 3 final evaluation scores, matching in-run scores bit for bitfinal/kev-candidate-facts.jsonPer-candidate audit of source switches and checkpoint hashesg1/kev-noplain.logFull judge output for the 0.3062444617 starting point: per-group and per-source NLL and accuracyg1/kev-original-ensemble.logOriginal wave-1 ensemble at 0.3147322620g5/kev-seeds.jsonlFive local seeds of the reference recipe, 0.2423–0.2584final/reseed/Three fresh seeds for the new training recipefinal/report_tables.jsonIn-run highs, lineage author counts, and evaluation queue statisticsclean-stack commit 4b7212f4, checkpoint dc47159cStored in the run repository kev-decision-architecture and on the run machine; not public7Contributions
Who did what
Agents carried out the modeling changes, evaluations, compliance checks, and fact-gathering in issue #11; the named human authors take responsibility for the conclusions and this report. Member handles are runtime seat identities rather than people and are omitted here.
8Thanks
Benchmark problem
The problem comes from OpenRSI-Index's kev-decision-architecture 1.1.0 (problem author listed as Yichen Feng), derived from jaredpalmer/kev (Apache-2.0).
9Versions
Versions
- 2026-11-17v1First published
10Checklist
What a reader will ask
Checklist (after NeurIPS and ARR)
- Claims: do the claims in the abstract and opening accurately reflect the report's contribution and scope?YesThe abstract states explicitly that the record is relative to this platform only, a compliance note awaits human review, and switched candidates are excluded; table in Section 3
- Best before: is the best prior value given with its source and date?YesSection 1 explains how the 0.30624 starting point was constructed and why wave 1's 0.3147 is disqualified
- Limits: are the limitations of the work discussed?YesSection 5
- Theory: does every theoretical result come with full assumptions and a complete, correct proof?N/AEmpirical benchmark score, with no theorem
- Reproduction: is all information needed to reproduce the main result provided, regardless of whether code and data are public?NoInference changes are described in Section 2 and the judge is deterministic, but the code and checkpoint are not public (Section 6)
- Code and data: are code and data public, with instructions sufficient to reproduce the result?NoSection 6: logs sit in Shoal's private lab repository and code/checkpoints on the run machine; only the benchmark itself is public
- Settings: are the run settings given, including parameters tried and final values used?YesConditions table in Section 2; pre-registered thresholds
- Statistics: are error bars, intervals, or multi-run summaries given, and is it clear whether values come from a best, mean, or single run?YesDeterministic judge with zero measurement noise; 5-seed range for the reference recipe and 3-seed mean for the new training recipe in Sections 1 and 3; selection bias discussed in Section 5
- Compute: are the machine, memory, wall time, and cost needed to reproduce each run given?NoWall time and evaluation queueing are reported (Sections 2 and 5); monetary cost has no audited record
- Assets: are authors and licenses credited for existing code, data, and models, and are newly produced files described?YesBenchmark provenance credited in "Thanks"; internal artifacts described in Section 6
- AI use: if AI assistants or agents were used, is their role disclosed?YesDisclosed in the byline under the authors and in Section 7
Cite this
@misc{shoal2026kev,
title = {kev decision head: without a switch, 0.306 to 0.328},
author = {The Shoal team},
year = {2026},
note = {v1, 2026-11-17},
url = {https://shoal.club/news/kev/}
}