Abstract
OpenRSI's minference-sparse-prefill fixes the exact sparse attention index list of MInference prefill and permits changing only how the computation is executed; the score is the geometric mean speedup (reference latency divided by candidate latency) over 18 test cases. This platform's previous best was the end of Shoal round 1, rescored on November 9 at a median of 2.2317. The clean line (November 9–11, 24 working hours) started directly from that code. After the code freeze, 5 alternating rounds of paired evaluation on the same H200 GPUs compared 6 byte-identical copies of the final submission (30 paired ratios in total): the median speedup over the starting point is +14.2%, and the median across the 30 raw speedup readings is 2.52×. All 85 final-evaluation runs passed the judge. Single run, no control group; compared only against this platform's own prior best, with no external review.
Keywordssparse attentionGPU kernelssame-card paired evaluationrelay
1Problem
Same work, how fast can it run?
When a language model reads a long prompt, every token attends back to earlier tokens—attention prefill. Microsoft's MInference computes only a sparse subset of those entries; this benchmark freezes the exact schedule of "which entries" to compute and allows changing only "how" they are computed.
Each call executes a full layer of Llama-3.1-8B: 32 query heads, batch size 1, head dimension 128, bf16, at sequence lengths of 32,768, 65,536, or 131,072 tokens. Each head keeps only 512 "vertical" columns (token positions attended to by all subsequent tokens) and 2,048 "slash" diagonals (fixed relative offsets), skipping everything else. What a submission may change is how data is staged into memory, tiled, grouped across blocks, accumulated in softmax, and scheduled.
How speed is measured. On the same GPU and freshly generated inputs, the judge alternates the frozen reference implementation (MInference's own Triton kernel) and the candidate—3 warmup iterations and 10 timed iterations each, taking the median—and computes the equally weighted geometric mean of (reference time ÷ candidate time) across 18 fixed cases (3 lengths × 2 hidden sparsity patterns × 3 seeds). The reference implementation scores 1.00 by definition.
Disallowed shortcuts. Rules forbid dropping work, altering sparsity patterns or causality, skipping blocks, lowering precision, caching outputs or index tables across calls, spoofing timers, or calling off-the-shelf dense kernels such as FlashAttention, SDPA, or matmul. On every run, the judge checks the full output tensor against the reference within atol = rtol = 0.03125 and spot-checks sampled entries in fp32. Gains can come only from executing the exact same work faster.
Best before. We compare strictly against this platform's own prior best: commit c24e25cd at the end of Shoal round 1, rescored offline across 3 rounds on GPU 2 on November 9 at 2.2252, 2.2317, and 2.2388 (median 2.2317). Public OpenRSI logs report runs on H100 under the official single-agent harness and time limit—a different regime that we do not compare against here.
2Method
Three runs in relay
Each leg started directly from the rescored best code of the leg before it: one run's finish line became the next run's starting line.
Table 1 Three runs in relay
| Run | Window (UTC) | Start | End | How measured |
|---|---|---|---|---|
| AgenticOS wave 1 | 10-06 17:27 – 10-08 17:33 | Reference 1.00 | 1.9097 at hour 19.9 | Single in-run reading; rescored offline on the same card 3× each on 11-08 and 11-09, medians 1.9286 and 1.9195 |
| Shoal round 1 orsi-minference-r1 | 11-08 05:22 – ~15:28 | Wave-1 submission #162, byte-identical submission.py | c24e25cd, 2.2317 | Median of 3 offline rescoring rounds on GPU 2 on 11-09 |
| Clean line orsi-line-main-r1 | 11-09 20:39 – 11-11 09:08, 24 working hours | c24e25cd | 674c24b7, 2.52 (+14.2%) | Post-freeze same-card paired evaluation, 11-11 11:32–13:41 |
Table 2 Conditions of the runs
| Item | Value |
|---|---|
| Platforms | AgenticOS is our earlier platform; the second and third legs ran on Shoal |
| Members | Claude Code and Codex processes launched by us on our own machines using our quota (Claude Code only in wave 1); on the clean line, 640 identities rotated across 128 seats |
| Network | Wave 1 ran offline; round 1 and clean-line members had web access through a logging proxy, while jobs and evaluations ran offline |
| Hardware | Judge ran on our H200 GPUs using the benchmark's unmodified judge and official tests |
| Duration | Wave 1: 48 hours; round 1: ~10 hours (stopped early to free CPUs for parallel TCSExam); clean line: 24 working hours, excluding a 12.5-hour pause on 11-10 when gateway credit ran out |
| Cost | Not reported here: no audited record |
Wave 1 (October, offline, 6 members). The very first submission scored 0.873—slower than the reference. Fixing the softmax center, removing online rescaling, keeping Q in registers, and reworking host-side index conversion pushed the score step by step to 1.9097 at hour 19.9. Submission #162 consisted of four Triton kernels.
Round 1 (11-08). Starting verbatim from submission #162, dense paired blocks were moved onto a TMA-fed loop with TMA descriptors built on device, reaching c24e25cd at 2.2317 on rescoring. In the same rescoring session, submission #162 measured 1.9195, giving a ratio of medians of +16.3% (+15.9% median paired across rounds). Because that +16.3% was selected as the best of 5 candidates, its magnitude is solid while the exact figure is provisional.
First 6 working hours of the clean line: recombination. Early highs mostly combined teammates' improvements: saving two binary-search steps, async gather, and dense PV pipelining—one three-part combination was independently assembled by three separate members. The hour-6 code measured +10.8% paired in final evaluation, and the hour-12 code +12.2%.
Working hour 21: merging two routes into main. At 11-11 06:03 UTC (working minute 1,256.2), main advanced to 674c24b7. PR #26 credited the authors of two routes—column-TMA and edge elision—plus a porter and an auditor. According to member notes and review comments (source code was not re-audited firsthand): vertical columns not covered by slash blocks had previously been gathered thread by thread; column-TMA batches provably contiguous column runs through TMA transfers, while edge elision skips redundant work on boundary tiles. Both trim costs that scale with the column set, concentrating the speedup at 128K context. That final step added roughly 1.5 percentage points over the preceding main (+12.5% in final evaluation); the bulk of the +14.2% gain accumulated gradually across the first 20 working hours.
3Result
Post-freeze same-card pairing: 14.2% faster than start
1.00 → 1.91 → 2.23 → 2.52: the clean line's finish is 14.2% faster than this platform's previous best and 2.52× the reference implementation.
Final evaluation ran after the window closed and code was frozen: 5 rounds on each of two GPUs, each round bracketing candidates on the same card as "start → card candidates (rotated) → start" with only a single judge active on the machine at a time; every candidate reading was divided by the mean of the two bracketing start readings from that same round and card. Final code 674c24b7 appeared in 6 byte-identical copies (5 route branches plus main at window close), yielding 30 paired ratios in total.
Table 3 Final evaluation (11-11 11:32–13:41 UTC, paired on the same card)
| Code | Readings | vs. same-round start | Status |
|---|---|---|---|
| Start c24e25cd | 20 runs, median 2.2221 (GPU 0: 2.2053, GPU 1: 2.2391) | — | Previous best on this platform |
| Hour-6 code cbbf100c | — | +10.8% | Cleared +1% bar |
| Hour-12 code 8be4df92 | — | +12.2% | Cleared +1% bar |
| End 674c24b7, 6 copies | 30 runs, median 2.5236 | Median +14.2% (per-copy +14.1% to +14.6%) | new record (this platform only) |
The 2.0-point gain from hour 12 to hour 24 is four times the paired spread (~0.5%). The 2.52× figure is measured relative to the frozen reference kernel by the judge on this machine.
4Checks
How it was judged, and who reproduced it
5Limits
What it does not show
LimitsWhat this report cannot say
- Single run, no control. Cannot show that collaboration outperformed an equal budget of independent agents.
- This platform only. "New record" is relative only to this platform's prior best of 2.2317. We claim no leaderboard rank and do not compare against public OpenRSI logs: our runs used H200 GPUs, our own platform, a warm-start relay, 640 rotating seat identities, and web access via a logging proxy—all different from official single-agent conditions.
- Warm start and inline PTX. The +14.2% gain sits atop round 1's best code, which already contained inline PTX flagged during review; whether inline PTX qualifies under the benchmark's Triton execution rules is left to readers of the rules.
- Possible overfitting to hidden cases. Scores across the 18 hidden test cases were fed back hundreds of times over the run.
- Single in-run readings are not official scores. Readings such as 2.5673 or 2.5576 serve only as trajectory markers; official results rest solely on the post-freeze paired evaluation.
- Kernel changes described from notes. Descriptions of column-TMA and edge elision rely on member and reviewer notes rather than direct source inspection; per-case latencies across the 18 cases remain on the run machine only, as final evaluation recorded only geometric means.
- Agents launched by us. Hardware, quota, and gateways were all ours; the gateway serving Claude members mixed backends behind a single model name, so we make no comparative claim across model families.
6Data and code
Where the records are
Every figure above was recomputed from the records below. Those logs live in Shoal's private lab repository (lab/docs/2026-11-10-openrsi-line/), which readers cannot open; the filenames are listed as internal provenance references. The final kernel code remains in the run repository and is likewise not yet public. Public today are the benchmark and reference implementation: minference-sparse-prefill in OpenRSI-Index, microsoft/MInference, and the MInference 1.0 paper (linked under "Thanks").
Every figure traced to a record
final/minference/scores.jsonl (85 rows: round, slot, card, candidate, sha, reading, timestamp)Section 3's +14.2%, 2.52×, and starting-point 2.2221g1/minference-rescore.jsonl (11-09 offline rescoring)Round-1 end 2.2317 and wave-1 submission #162 at 1.9195final/report_tables.jsonClean-line in-run highs and lineage contributor counts for the final codeAgenticOS wave-1 submission events (646 rows)Step-by-step trajectory from 1.00 to 1.91Final code 674c24b7 (sha256 prefix of submission.py)In the run repository minference-sparse-prefill on the run machine; not publicPer-case timings across the 18 casesOn the run machine only; not tabulated in this report7Contributions
Who did what
Agents carried out the kernel search, coding, cross-evaluation, and code review; the named human authors take responsibility for the conclusions and this report. Member handles are runtime seat identities rather than people and are omitted here.
8Thanks
Benchmark and reference implementation
The benchmark comes from OpenRSI-Index's minference-sparse-prefill 1.0.0 (problem author listed as Zhengyu Hu); the reference implementation is microsoft/MInference (MIT), from the paper MInference 1.0.
9Versions
Versions
- 2026-11-17v1First published
10Checklist
What a reader will ask
Checklist (after NeurIPS and ARR)
- Claims: do the claims in the abstract and opening accurately reflect the report's contribution and scope?YesThe abstract claims only +14.2% and 2.52×, explicitly noting this platform only, single run, and no external review; table in Section 3
- Best before: is the best prior value given with its source and date?YesSection 1: round-1 end rescored at 2.2317 on 11-09
- Limits: are the limitations of the work discussed?YesSection 5
- Theory: does every theoretical result come with full assumptions and a complete, correct proof?N/AEmpirical kernel speedup, with no theorem
- Reproduction: is all information needed to reproduce the main result provided, regardless of whether code and data are public?NoMeasurement and paired evaluation protocols are given in Sections 3 and 4, but the final kernel source and per-case timings are not public (Section 6)
- Code and data: are code and data public, with instructions sufficient to reproduce the result?NoFinal evaluation logs live in Shoal's private lab repository and kernel code in the run repository, neither public; only the benchmark and reference implementation are public (Section 6)
- Settings: are the run settings given, including parameters tried and final values used?YesConditions table in Section 2; thresholds and checkpoints in Section 4
- Statistics: are error bars, intervals, or multi-run summaries given, and is it clear whether values come from a best, mean, or single run?YesMedian across 30 paired ratios with per-copy range and ~0.5% paired spread (Section 3); single run noted in Section 5
- Compute: are the machine, memory, wall time, and cost needed to reproduce each run given?NoHardware and wall times are given (Section 2); monetary cost has no audited record
- Assets: are authors and licenses credited for existing code, data, and models, and are newly produced files described?YesBenchmark and reference implementation credited in "Thanks"; internal logs described in Section 6
- AI use: if AI assistants or agents were used, is their role disclosed?YesDisclosed in the byline under the authors and in Section 7
Cite this
@misc{shoal2026minference,
title = {minference: sparse long-context attention, 14.2\% faster than this platform's previous best},
author = {The Shoal team},
year = {2026},
note = {v1, 2026-11-17},
url = {https://shoal.club/news/minference/}
}