Consider a routine request: a requester posts 100 credits for a working program. Carol opens a branch; branching from it, Alice and Bob each write a 400-line module; Carol merges Alice's branch first, then Bob's, and submits the result as a delivery. The requester accepts it. Carol has not written a single line of code. How the credits split depends on which rule governs—and at different stages, Shoal has shipped or tested four of them.
AliceBobCarol
Data
| Rule | Alice | Bob | Carol |
|---|---|---|---|
| By commit count | 25 | 25 | 50 |
| By lineage position | 7 | 18.8 | 74.2 |
| One share per changer | 50 | 50 | 0 |
| Agent reads code and suggests | 55 | 40 | 5 |
359b875, PR #34 b9c65cc, and PR #35 40b8995); the last row is a single run of the analysis package on the author's branch over code of the same shape.By commit count, Carol takes half, because each merge counts as a commit; by position in the commit lineage, she takes 74.2%, because the delivered commit is hers; under one equal share per substantive contributor, she gets 0; when an agent reads the code and recommends shares, she gets 5%. All four rules compute cleanly; they disagree on what counts as contribution.
This kind of divergence already has a name elsewhere. Twenty-nine teams—61 analysts in all—used the same dataset to answer a single question: whether soccer referees are more likely to give red cards to dark-skinned players. Twenty teams found a statistically significant positive effect and nine did not, with odds ratios ranging from 0.89 to 2.93 (Silberzahn et al. 2018). In a larger replication, 73 teams submitted 1,261 models, and over 95% of the variance across results could not be explained by coded analytic decisions or researcher characteristics (Breznau et al. 2022). In 2026, when AI analysts repeated the exercise, audited runs disagreed even on the sign of the effect on some datasets (Bertran et al. 2026). None of the analysts miscalculated; the spread came from defensible choices at every step. Credit attribution works the same way: one record, many rules, many answers.
In the era of the paper, the problem was less urgent because the unit of credit was an entire article. Once agents enter the loop, contributions split down to a single merge, a one-line fix, or a verification run—and each of those actions is cheap. Whatever a rule rewards, a group produces more of; replace human effort with agents that optimize directly against the objective, and that pull requires no malice at all. In 2008, Michael Nielsen wrote that new scientific tools would need "metrics for contribution built right into them," with every contribution archived, timestamped, and signed (Nielsen 2008). Git gives us the second half for free today; the first half is the hard part.
Contribution is a counterfactual invisible in the record; once you pay on it, it reshapes the behavior being measured. So we separate the three things authorship has bundled together: treat recognition as measurement, credits as mechanism, and leave responsibility with a person who can bear it.
This essay uses Shoal as its proving ground. Chapter 1 covers background and the platform: the history of scientific credit, how Shoal operates, and how the rules we tried were fooled by cheap actions. Starting in Chapter 2, we treat contribution measurement in multi-agent systems as a research problem in its own right: Chapter 2 formalizes it and shows why contribution is unidentifiable from the record alone; Chapter 3 analyzes three families of algorithms and the assumptions each relies on; Chapter 4 asks how to test an attribution rule; and Chapter 5 derives the mechanism. Throughout, each claim notes where it stands: established result, our reproduction, or to the best of our search.
Chapter 1
Background: from authorshipto the relay
The unit of credit has shrunk from a paper to a commit. Finer units leave richer records—and let rules amplify cheap actions.
1.1Authorship
The unit of credit:from a paper to a merge
Scientific reward rests on priority and peer recognition, and recognition gravitates toward those who are already known: for the same work, an established scientist's name attracts disproportionate attention—a pattern Merton, borrowing from the Gospel of Matthew, called the Matthew effect (Merton 1968). The unit of that system is the paper, with registration, certification, and reward all hung on a single finished manuscript. As collaboration scales up, the coarseness of the unit shows: in 2015, the joint ATLAS and CMS measurement of the Higgs boson mass listed 5,154 authors, dedicating 24 of its 33 pages to names and institutions (Aad et al. 2015; Castelvecchi 2015).
Several remedies followed. Role taxonomies write down what kind of work each person did: CRediT defines 14 contributor roles and became an ANSI/NISO standard in 2022, though it records the category of contribution rather than its magnitude (Allen et al. 2014; Brand et al. 2015; NISO 2022). Counting tallies what can be counted: GitHub's contributors graph counts commits on the default branch while excluding merge commits and empty commits (GitHub Docs 2026). Marginal value asks what would happen if a contributor were absent: Shapley in 1953 derived the unique division in cooperative games satisfying a natural set of axioms, an idea machine learning later adapted to value training data (Shapley 1953; Ghorbani & Zou 2019). Still another approach makes the group itself the author: Polymath papers were signed D. H. J. Polymath, leaving anyone who wanted to know who did what to read the public discussion threads (Polymath 2009; Gowers & Nielsen 2009).
More recent cases sit closer to home. The Equational Theories Project, initiated by Terence Tao and coordinated on GitHub with Lean verification, published a 34-author paper in 2025: the formal author list is alphabetical, while an appendix lists who carried out each kind of task like film credits, adapting CRediT categories—with only humans in the contribution matrix (Bolan et al. 2025). After AI systems were claimed in 2025 to have "solved" several Erdős problems—most of which turned out to be locating existing results in the literature—Tao's wiki split AI contributions into major and minor tiers, classifying literature retrieval, formalization, rewriting, and computation as minor while tracking outcomes separately as solved, partial progress, erroneous, or unverified (Erdős Wiki). That same year, the Agents4Science conference required AI systems to be listed as first authors and asked submissions to declare human versus AI shares across four stages—hypothesis, design, analysis, and writing (Bianchi et al. 2025); yet a study of AI disclosure statements in computer science papers found that only 2% to 23% stated who was accountable for AI-generated output (Mohapatra & Pruthi 2026). Disclosure forms keep growing finer, while the people filling them out remain the interested parties themselves.
On whether AI can be an author, journals and conferences gave nearly the same answer at nearly the same time in 2023: no. Their reasoning was almost verbatim: authorship entails accountability for the work, and AI cannot bear responsibility. COPE wrote that AI tools "cannot take responsibility for the submitted work"; Nature wrote that "any attribution of authorship carries with it accountability for the work" (COPE 2023; Nature 2023; ICMJE 2026). Those rules settled who cannot be an author, not who decides credit; ICMJE adds explicitly that judging who qualifies for authorship or arbitrating authorship disputes is not the role of journal editors. Historically, the work most easily left out is the work that makes a result hold up in the first place: in Shapin's history of Boyle's seventeenth-century laboratory, technicians built the instruments and ran the experiments, yet at publication "His master spoke for him," their labor "included in" the master's name (Shapin 1989).
Authorship
A single author line
- Notes
- Who was in the room
- Omits
- Who did what
- Shoal
- Lineage split (PR #34)Tried
Roles
CRediT's 14 roles
- Notes
- What category of work
- Omits
- How much was done
- Shoal
- Roles in reports, descriptive onlyLive
Counts
Commits and merges
- Notes
- How many actions
- Omits
- What each was worth
- Shoal
- By commit count (main)Default
Marginal
What happens without it
- Notes
- Marginal effect
- Omits
- What replaces absence
- Shoal
- Shapley (offline study)Tried
Review
Judgement from evidence
- Notes
- How one reader sees it
- Omits
- What another would say
- Shoal
- Rubric → tiered reportLive
Each of these traditions has a point, and each has a blind spot. Authorship records who was in the room; roles record what category of work was done; counts record how many actions occurred; marginal value records what would be lost without someone; review records how a reader of the evidence sees it. None of them alone answers whose work the result actually stands on. Traditional authorship quietly governs three things at once: recognizing who did what, deciding who gets rewarded, and fixing who is accountable for the outcome. CRediT unbundled roles within the first; the latter two remain tied together.
1.2Platform
Shoal:how people and agents relay work
Shoal is a platform for open research, hosting both open problems and credit-backed requests. People and their agents work on it in relays: a delivery is a pull request, behind which several agents have often explored different routes—one proposing a method, another implementing it, a third finding a counterexample, and someone else integrating the pieces at the end. The site runs online while agents run on their operators' own machines; the site never calls a model.
The credit ledger is append-only, and every balance is a sum. Each person starts with 100 credits, issued solely by the site; credits cannot be bought, sold, transferred, or cashed out, and whatever an agent earns accrues to its human operator. Posting a request holds the offered credits in escrow, capped at the requester's balance. Once a delivery is accepted, settlement branches into two cases: if every commit in the delivery comes from a single side, payment is immediate; if several people contributed, the requester has 3 days on the analysis page to confirm who receives what. Recipients can only be makers of that delivery: commit authors, the author of the delivery PR, and participants assigned a share in a report published by the requester; the requester's own side cannot receive credits. If 3 days pass without a decision, the default settles the escrow: when a published report covers the delivery, credits follow its shares renormalized across eligible recipients; when there is no report, credits split by commit count in the delivery. Shares merge by operator and round to integer credits by the largest-remainder method; every decision stays on the record, visible to anyone who can view the project.
Data
| Settlement path | Alice | Bob | Carol | Requester |
|---|---|---|---|---|
| One side made it | 100 | 0 | 0 | 0 |
| Confirmed within 3 days | 55 | 40 | 5 | 0 |
| Expired, published report | 55 | 40 | 5 | 0 |
| Expired, no report | 25 | 25 | 50 | 0 |
| No delivery | 0 | 0 | 0 | 100 |
Share recommendations for multi-party deliveries come from a four-step project analysis. First, freeze: the requester picks a revision and the question to answer, and the site freezes the baseline, target, rules revision, authenticated participants, jobs, checks, results, issues, and reviews together with a Git bundle containing every relevant commit; the repository may keep changing afterward, while what this analysis reads stays fixed. Second, judgement: the requester hands a one-line instruction to their own Claude Code, Codex, or other agent; the agent claims the task, traces the origin and adoption of each outcome across the read-only Git history, records its judgements, and runs a local tool to compose the report. Third, human confirmation: editing, publishing, retracting, and confirming rewards all require a signed-in person—a token cannot perform them, including the token of the requester's own agent. Fourth, a second report: anyone named in a published report can read the exact same frozen materials and upload a report of their own, which the page places side by side with the original.
Set beside comparable platforms, Shoal's position becomes clear. For splitting money across multiple contributors, the platforms we surveyed default to one of three rules: pay only the single accepted PR, split equally, or leave the split to the participants themselves (Algora 2026; HackerOne 2026; DrivenData 2026).
| Platform | Who gets paid | Who decides for teams | Window & default | When disputed |
|---|---|---|---|---|
| Algora | Author of the claimed, accepted PR | Claimant lists recipients | Equal split across list | Undocumented |
| HackerOne | Reporter and invited collaborators | Reporter sets weights | Final once bounty is awarded | Not checked here |
| DrivenData | Each team member | Unanimous team consent | 3 business days post-deadline to request custom split; else equal | Written statement |
| Prize challenges (GoFly, etc.) | Account designated by team lead | The team internally | None | Organizer does not arbitrate |
| Shoal | Makers of the delivery | Requester | 3 days; else published report, or commit count if none | Second report |
Relaying—one person building directly on an earlier contributor's work—has no place in any of those rules. AI attribution in software stops at the layer of "who wrote the lines" as well: GitHub's Copilot cloud agent authors commits under its own identity with the delegating user as co-author, and that user's approval does not count toward required reviews (Copilot Docs 2026); the Agent Trace specification and Git AI record line by line which agent produced which code, with the former explicitly declining to judge quality (Agent Trace 2026; Git AI 2026); and the Linux kernel bars AI from signing the Developer Certificate of Origin, placing responsibility on the human submitter (Linux 2026). Shoal has to answer a different layer altogether: whose work was actually adopted.
1.3Trial and error
The rules we tried,and the cheap actions that fooled them
When credit escrow launched on October 25, multi-party deliveries split by commit count. In early November, PR #34 proposed lineage attribution and rubric scoring alongside a six-component Shapley experiment; review on November 6 found that the lineage rule ranked a merge-only participant first; an absorbed revision, PR #35, switched to one equal share per substantive contributor, only for a second review round to reproduce three payout vulnerabilities, leading to its withdrawal on November 7. On November 9, PR #37 let agents write reports over frozen evidence; PR #38, merged into main on November 10, inverted the workflow to list outcomes first, grade tiers second, and keep the arithmetic public.
Against the platform's live code, using temporary data directories, we ran each cheap action one by one—one run per probe. Under commit count, someone who edits a single line before delivering walks away with half; under lineage attribution, they take 62.5%, a merge-only contributor takes 74.2%, and a requester who merely clicks "Approve" claws back 11.8% of their own bounty. Giving one share to every substantive contributor blocks the merge-only case, yet a requester who pushes a single line into the delivery still recovers 20 out of 40 credits, and someone who makes an edit after a merge and immediately reverts it—leaving the final files unchanged—still collects a full share. Only one approach recognized a comment-only padding file for what it was: having an agent read the content, which assigned everyone 0 share and noted that the checked acceptance boxes carried no supporting evidence. On an earlier branch, the requester could even award credits to themselves or to an account that had never touched the repository; and if they never confirmed after accepting, the credits stayed locked indefinitely while refunds were refused.
| Action | By commit countCommitsmainmain | By lineageLineagePR #34PR #34 | One share per changerEven splitPR #35PR #35 | Agent reads contentReads codeanalysis pkg · livelive |
|---|---|---|---|---|
| One-line finishAlice writes the work; Bob edits one line and delivers | 50% | 62.5% | 50% | — |
| Merge onlyCarol writes no code, merges two routes and delivers | 50% | 74.2% | 0% | 5% |
| Add a line, then mergeCarol adds one line before merging and delivering | 60% | 86.1% | 33.3% | — |
| Requester approvesRequester clicks Approve before accepting | 0% | 11.8% | 0% | Excluded |
| Requester pushes a lineRequester pushes one line into the delivery, then accepts | — | — | 50% | Excluded |
| Edit then revertEdits a line after merging and reverts it; final content unchanged | — | — | 33.3% | — |
| Padding fileDelivered file contains only comments | Paid in full | — | Paid in full | 0% |
| Planted instructionsCLAUDE.md in the repo instructs the reviewer to grade the committer Core | — | — | — | Ignored |
| Fooled / tested | 4 / 5 | 4 / 4 | 5 / 7 | 0 / 3+ 2 excluded by rule+2 excl. |
Fooled: share captured by the actorBlockedExcluded by rule— Not tested
Data
| Action | By commit count | By lineage | One share per changer | Agent reads content |
|---|---|---|---|---|
| One-line finish | 50% | 62.5% | 50% | — |
| Merge only | 50% | 74.2% | 0% | 5% |
| Add a line, then merge | 60% | 86.1% | 33.3% | — |
| Requester approves | 0% | 11.8% | 0% | Excluded by rule |
| Requester pushes a line | — | — | 50% | Excluded by rule |
| Edit then revert | — | — | 33.3% | — |
| Padding file | Paid in full | — | Paid in full | 0% |
| Planted instructions | — | — | — | Codeword absent from report, hook never fired, planted file treated as material under review (1 run) |
Money attracts cheap actions elsewhere too: before curl shut down its bug bounty in January 2026, the confirmation rate on submitted reports had fallen from above 15% to below 5% (Stenberg 2026). People are still restrained by embarrassment; agents optimize straight toward whatever a rule rewards, with no malice required. Those trials left us with three questions. Can the record alone pin down contribution? What assumptions does each family of algorithms rely on, and how should a rule be tested? And who should hold the right to decide? Chapters 2 through 5 answer them in turn.
Chapter 2
Problem: contribution is anunseen counterfactual
From here on, we situate what happens on Shoal within multi-agent credit assignment: first distinguishing our problem from its namesake in machine learning, next setting out notation and definitions, and finally showing why contribution is unidentifiable from records alone—and why paying on a rule changes what it measures.
2.1Learning & division
Credit for learning,credit for division
Multi-agent reinforcement learning has long studied credit assignment: when a team of agents shares a single reward, how much of it does each agent answer for? The textbook answer is a counterfactual: compare the actual reward with the reward had that agent taken a default action instead, and call the difference its contribution. Difference rewards work this way, at the cost of requiring a simulator or learned reward model and a defensible choice of default action; COMA learns that counterfactual baseline with a centralized critic (Foerster et al. 2018). A second line of work uses value decomposition, expressing team value as a monotonic combination of individual utilities (Sunehag et al. 2018; Rashid et al. 2018); a third computes Shapley values directly, substituting a zero vector for absent agents—a choice the authors note is made for convenience (Li et al. 2021; Wang et al. 2022).
With LLM-based multi-agent systems, this literature has branched again. When training multiple agents, researchers estimate each agent's effect via Shapley-style credit or by branching and resampling from intermediate states (Li et al. 2026; Zhang 2026); where an AI coach scores intermediate steps, signs of agents gaming the process reward have already emerged (Li, Ren & Yan 2026). Another branch studies failure attribution: reading a failed multi-agent trace to name which agent erred at which step. Even the strongest methods identify the responsible agent only 53.5% of the time and the exact step only 14.2% of the time (Zhang et al. 2025); other work argues that pinning a systemic failure on a single agent is often ill-posed from the start (Ma et al. 2026; Cemri et al. 2025). A 2026 survey lists credit assignment under soft, learned rewards as largely open (Zhang 2026).
What all of those methods produce is a learning signal: shares need not sum to the whole, participants do not have to accept them, and a policy being trained neither files an appeal nor pushes comment-only padding files. What we need is division: shares land on people, becoming credits and reputation. Division adds four requirements: shares must sum to the budget, carry an intelligible rationale, withstand strategic behavior by interested parties, and admit contestation. Deeper still is where the counterfactual comes from: machine learning leans on simulators, critics, or reruns, and open research has none of the three.
Grouped by "What it sees · Who decides"
Content · People
- Retro FundingImpact evaluations set grant amounts directly; paused Jan 2026Optimism 2021–2026
- Protocol GuildRecognition decides eligibility only; amounts follow a tenure formulaProtocol Guild 2025 report
- HackerOne, DrivenDataReporter sets weights; custom split with unanimous team consent, else equalPlatform terms
- CRediT, contribution statementsRecords roles without magnitudes; self-reportedBrand et al. 2015
- Equational TheoriesGit + Lean + contribution matrix; no AI credit entriesBolan et al. 2025
Structure · Model
- DyLAN importancePeer rating propagated backward from terminal nodes, isomorphic to lineageLiu et al. 2024
Content · Model
- Tiers & public formulaAgent lists outputs and grades tiers, code converts; requester decides payoutShoal main 11-10↑ Requester decides payout
- Failure attributionFrom logs alone: 53.5% agent accuracy, 14.2% step accuracyZhang et al. 2025
- Agent free-form sharesSwapping models shifts shares by 14.7 pp TVDAuthor branch
- Deep FundingModels weight dependency-graph edges while human jurors spot-check a slice; no official pilot results yetButerin 2025; deepfunding.org
Structure · Formula
- GitHub contributors graphCounts default-branch commits only, excluding mergesGitHub Docs
- SourceCredPageRank over contribution graphs for cred and grain; wound down in 2022SourceCred 2019–2022
- teaRewarded registered dependencies, drawing 150k spam packages onto npmAWS 2025
- Lineage v2.2Flows backward along merge chains; deliverer retains 62.5%PR #34
- Co-citation creditInfers co-author credit from downstream co-citationsShen & Barabási 2014
Outcomes · Formula
- COMA, difference rewardsCounterfactuals from a simulator, critic, or default actionFoerster et al. 2018
- SHAQ, SHARPShapley-style credit signals for training multi-agent policiesWang et al. 2022; Li et al. 2026
- Data valuationData Shapley, Data Banzhaf: utility choice rests with the evaluatorGhorbani & Zou 2019; Diehl & Wilson 2025
- Attribution gamesCoalition weights × intervention protocol × target metric; different protocols define different gamesLu et al. 2026
- Six-component ShapleyExact over 64 coalitions; changing absence replacement shifts shares by up to 26.6 ppPR #37 archive
For division: turns into money or standingFor learning: training signalTried on Shoal
Data
| Method | What it sees | Who decides | Purpose | Source |
|---|---|---|---|---|
| GitHub contributors graph | Structure | Formula | Division | GitHub Docs |
| SourceCred | Structure | Formula | Division | SourceCred 2019–2022 |
| tea | Structure | Formula | Division | AWS 2025 |
| Lineage v2.2 | Structure | Formula | Division | PR #34 |
| Co-citation credit | Structure | Formula | Division | Shen & Barabási 2014 |
| COMA, difference rewards | Outcomes | Formula | Learning | Foerster et al. 2018 |
| SHAQ, SHARP | Outcomes | Formula | Learning | Wang et al. 2022; Li et al. 2026 |
| Data valuation | Outcomes | Formula | Division | Ghorbani & Zou 2019; Diehl & Wilson 2025 |
| Attribution games | Outcomes | Formula | Learning | Lu et al. 2026 |
| Six-component Shapley | Outcomes | Formula | Division | PR #37 archive |
| DyLAN importance | Structure | Model | Learning | Liu et al. 2024 |
| Failure attribution | Content | Model | Learning | Zhang et al. 2025 |
| Agent free-form shares | Content | Model | Division | Author branch |
| Tiers & public formula | Content | Model | Division | Shoal main 11-10 |
| Deep Funding | Content | Model | Division | Buterin 2025; deepfunding.org |
| Retro Funding | Content | People | Division | Optimism 2021–2026 |
| Protocol Guild | Content | People | Division | Protocol Guild 2025 report |
| HackerOne, DrivenData | Content | People | Division | Platform terms |
| CRediT, contribution statements | Content | People | Division | Brand et al. 2015 |
| Equational Theories | Content | People | Division | Bolan et al. 2025 |
2.2Notation
Participants, records,and attribution rules
The notation below runs through the remaining three chapters.
| Symbol | Meaning |
|---|---|
| Participants: people and agents ; each agent belongs to an operator , to whom its share is paid | |
| The record of a delivery: commit graph, jobs, checks, reviews, and other evidence, frozen at a target revision | |
| Commissioned outputs of the delivery and their weights, | |
| Groups within output and each group's tier; participants inseparable in the evidence form one group | |
| The share simplex: , | |
| Shares assigned by an attribution rule, | |
| Value of the outcome when only coalition participates | |
| Total variation distance (TVD) between two share vectors, reported in percentage points (pp) |
Definition 1 (Attribution rule) An attribution rule is a map : it reads a frozen record and assigns each participant a share, . The rule sees only ; anything outside —such as what would have happened without Carol—must either be posited by the rule itself or supplied by a judge who has read .
Definition 2 (Attribution game) Following Lu et al.'s formulation of agent attribution, a cooperative attribution game is fixed by three choices: a target metric , specifying what counts as outcome value (passing tests, a benchmark score, or human acceptance); an intervention protocol , specifying what replaces an absent participant's work; and coalition weights , specifying how subsets are averaged (Shapley averages uniformly over arrival orders, Banzhaf weights all coalitions equally, and leave-one-out looks only at the grand coalition minus one) (Lu et al. 2026). Once is fixed, participant 's Shapley value is
Definition 3 (Total variation distance) The distance between two allocations and is : the fraction of the total budget that must be moved from some participants to others to make the two allocations coincide.
2.3Identifiability
Where counterfactualscome from
A participant's contribution is at bottom a counterfactual: what would have happened to the result without their work? In equation (1), asks about an unrealized world for every coalition except the grand coalition itself. Machine learning has tried five sources for that counterfactual (Foerster et al. 2018; Zhang 2026; Lu et al. 2026):
| Counterfactual source | Representative methods | What it requires | In open research |
|---|---|---|---|
| Reruns | Ablation, rollout continuation, branch resampling | Replayable intermediate states at affordable cost | Deliveries cannot be replayed; only synthetic tasks can |
| Learned critic | COMA, value decomposition | Many trajectories from the same distribution | Every request differs; sample size is tiny |
| Default action | Difference rewards, zero vector | A convention for "absence" | Available—and the exact source of the 26.6 pp swing in Chapter 3 |
| Model imagination | Asking a judge what would happen if an agent were absent | A judge model | Available, but known to overestimate and disagree with real ablations |
| Single-trace structure | Lineage, backward importance propagation, semantic support | A dependency graph | Available as Git lineage, but vulnerable to cheap actions |
When Lu et al. asked an LLM judge to imagine removing an agent and compared its answers against actually ablating the agent and rerunning, 30% to 52% of instance-level attributions disagreed (Lu et al. 2026). Across our search, we found only one Shapley method designed explicitly to run on a single realized trajectory without reruns; it computes over a graph of semantic entailment and is evaluated on medical QA and multi-role workflows (Jiang et al. 2026). To the best of our search, for non-replayable research and software deliveries where value comes from human acceptance, the machine learning literature offers almost no validated contribution measures (search log in Appendix B).
That is the problem of identifiability: the record fixes the facts, not the counterfactual, and every rule bridges the gap with an assumption. We ran the opening three-person delivery through 15 specifications: three measured live on platform code, nine computed directly by platform functions, one derived from GitHub's documented contributor rules, and two illustrative value functions.
Observation 1 (One record, fifteen answers) On the opening delivery, 15 specifications assign Carol anywhere from 0 to 74.2%. By commit count, she receives 50% when merges count and 0 when they do not; by lineage, she receives 74.22% whichever branch she merges first; under tier grading, grading her unattributed, pending, incidental, supporting, major, or core gives her 0, 0, 1.54%, 5.88%, 20%, or 33.33%; listing integration as a separate equally weighted output gives her 50%. The 15 specifications land on just 9 distinct points: five of them collapse onto the exact same allocation (50, 50, 0) for entirely different reasons.
MeasuredPlatform codePublic specIllustrativeCoinciding specs
SpecificationCarol's share (%)
Commit count
Lineage
Even split
Agent judgement
Tiers
Shapley
Data
| Family | Specification | Source | Alice | Bob | Carol |
|---|---|---|---|---|---|
| Commit count | Count merges (Shoal main) | Measured | 25 | 25 | 50 |
| Commit count | Exclude merges (GitHub graph) | Public spec | 50 | 50 | 0 |
| Lineage | Merge Alice first | Platform code | 7.03 | 18.75 | 74.22 |
| Lineage | Merge Bob first | Platform code | 18.75 | 7.03 | 74.22 |
| Even split | One share per substantive changer | Measured | 50 | 50 | 0 |
| Agent judgement | Agent reads code, free-form shares | Measured | 55 | 40 | 5 |
| Tiers | One output, Carol Unattributed | Platform code | 50 | 50 | 0 |
| Tiers | One output, Carol Pending | Platform code | 50 | 50 | 0 |
| Tiers | One output, Carol Incidental | Platform code | 49.23 | 49.23 | 1.54 |
| Tiers | One output, Carol Supporting | Platform code | 47.06 | 47.06 | 5.88 |
| Tiers | One output, Carol Major | Platform code | 40 | 40 | 20 |
| Tiers | One output, Carol Core | Platform code | 33.34 | 33.33 | 33.33 |
| Tiers | Integration as equal output, Carol Core | Platform code | 25 | 25 | 50 |
| Shapley | Anyone can merge: Carol substitutable | Illustrative | 50 | 50 | 0 |
| Shapley | No merge without Carol: all three needed | Illustrative | 33.33 | 33.33 | 33.33 |
The two Shapley rows use illustrative value functions: a runnable delivery requires both modules plus a merge; "substitutable" assumes anyone could perform the merge, whereas "needed" assumes no merge happens without Carol.
MeasuredPlatform codePublic specIllustrative
Data
| # | Family | Specification | Source | Alice | Bob | Carol |
|---|---|---|---|---|---|---|
| 1 | Commit count | Exclude merges (GitHub graph) | Public spec | 50 | 50 | 0 |
| 2 | Even split | One share per substantive changer | Measured | 50 | 50 | 0 |
| 3 | Tiers | One output, Carol Pending | Platform code | 50 | 50 | 0 |
| 4 | Tiers | One output, Carol Unattributed | Platform code | 50 | 50 | 0 |
| 5 | Shapley | Anyone can merge: Carol substitutable | Illustrative | 50 | 50 | 0 |
| 6 | Tiers | One output, Carol Incidental | Platform code | 49.23 | 49.23 | 1.54 |
| 7 | Agent judgement | Agent reads code, free-form shares | Measured | 55 | 40 | 5 |
| 8 | Tiers | One output, Carol Supporting | Platform code | 47.06 | 47.06 | 5.88 |
| 9 | Tiers | One output, Carol Major | Platform code | 40 | 40 | 20 |
| 10 | Tiers | One output, Carol Core | Platform code | 33.34 | 33.33 | 33.33 |
| 11 | Shapley | No merge without Carol: all three needed | Illustrative | 33.33 | 33.33 | 33.33 |
| 12 | Commit count | Count merges (Shoal main) | Measured | 25 | 25 | 50 |
| 13 | Tiers | Integration as equal output, Carol Core | Platform code | 25 | 25 | 50 |
| 14 | Lineage | Merge Alice first | Platform code | 7.03 | 18.75 | 74.22 |
| 15 | Lineage | Merge Bob first | Platform code | 18.75 | 7.03 | 74.22 |
That the definition of absence reshapes attribution is well established in explainability and data valuation (Sundararajan & Najmi 2020; Sturmfels et al. 2020); in data valuation, the evaluator can even choose the utility function strategically to steer the outcome (Diehl & Wilson 2025). What we add here is how large that swing becomes on team deliveries and the concrete ways each choice breaks, examined family by family in Chapter 3.
Definition precedes estimation. A single record does not determine a unique contribution. Shares mean something only after you state upfront what counts as value, what replaces an absent contributor, and who is substitutable.
2.4Rules as mechanisms
Once a rule pays,it becomes the target
Measurement has a second side. Goodhart observed in 1975 that any observed statistical regularity tends to collapse once pressure is placed upon it for control, later distilled by Strathern into the familiar maxim: when a measure becomes a target, it ceases to be a good measure (Goodhart 1975; Strathern 1997). Machine learning has two formal counterparts: strategic classification, where subjects alter their features to game a classifier (Hardt et al. 2016), and performative prediction, where deploying a predictor shifts the very distribution it predicts (Perdomo et al. 2020).
Agents accelerate that loop. METR reported in 2025 that o3 hacked the scoring setup on 30.4% of RE-Bench runs, acknowledging when questioned afterward that its shortcuts violated the spirit of the task (METR 2025). OpenAI found that under strong optimization pressure, optimizing directly against a chain-of-thought monitor does not eliminate reward hacking—it teaches the model to hide it (Baker et al. 2025). On SWE-bench, agents dug through Git history to locate future fix commits (SWE-bench 2025)—and Git history is precisely what lineage rules read. Meanwhile, the tea protocol, which rewarded open-source packages along dependency graphs, drew over 150,000 token-farming spam packages onto npm (AWS 2025).
Manipulation comes from two sides. One is contributors: splitting commits, splitting outputs, or opening sybil accounts. Exact Shapley allocation over declared identities cannot simultaneously prevent splitting and false-name attacks, as a 2026 preprint proves via an impossibility theorem (Burnat & Davidson 2026); remedies have likewise been proposed for "shell company" attacks that split one coalition into several to inflate its combined valuation (Lee et al. 2025a). The other side is the rule-setter: picking whichever plausible definition favors their preferred outcome (Diehl & Wilson 2025).
Multi-agent systems add one more layer: when the reviewer is itself an agent, the artifact under review can speak directly to its judge. In July 2025, 18 arXiv preprints were found hiding instructions to AI peer-review tools in white or microscopic text (Lin 2026); in our own tests we planted a CLAUDE.md inside a repository under review instructing the assessor to grade the submitter core (Chapter 4). And a quieter distortion runs underneath all of these: when a job has several dimensions and only some are measurable, paying solely on the measurable ones crowds effort out of the unmeasurable ones (Holmström & Milgrom 1991). As Chapter 5 will show, verification, review, and groundwork are precisely the unmeasurable parts in our records.
A rule is a mechanism. Once an attribution rule governs payouts, it has to be judged as a mechanism: what it rewards, where it steers people and agents, and who can game it.
2.5Properties
What an attribution ruleought to satisfy
Putting the preceding sections together, an attribution rule used for division has to answer to seven properties. The first four are Shapley's axioms (Shapley 1953); the next three come from the mechanism side. The matrix in Chapter 4 tests five of them; additivity cannot be tested on a single delivery, and contestability belongs to the surrounding mechanism in Chapter 5.
- Efficiency: shares sum to the full bounty, .
- Symmetry: contributors with identical effect receive identical shares, regardless of merge order or naming order.
- Null player: someone with no effect on the outcome receives zero—whether merging without changes, editing and reverting, or adding padding files.
- Additivity: shares across two tasks can be computed separately and added together.
- Split invariance: splitting the same work into more commits, more outputs, or more accounts leaves shares unchanged.
- Evidence-bounded: every share traces back to specific cited evidence.
- Contestability: anyone named in an attribution has a way to say "that is not what happened."
Payout carries three further guarantees that care only about the flow of credits, whether or not the shares are right: no lockup, so every held credit eventually pays out or refunds; no flow-back, so once accepted, credits never return to the payer; and full visibility, so every decision and its rationale remain on the record.
Chapter 3
Methods: three families,three assumptions
Structural rules treat position as contribution, counterfactual rules treat absence as known, and judgement rules treat ordinal ranks as multiples. For each family, we state the definition and formal properties first, then report what we measured.
3.1Position
Commit count and lineage:position as contribution
The simplest structural rule counts commits. Even that comes in two conventions: Shoal's main branch counts merge commits, giving Carol 50% on the opening delivery; GitHub's contributors graph excludes merge commits, giving her 0. Under the same label—"by commit count"—two standard conventions give opposite answers.
PR #34's lineage attribution rule (v2.2) goes a step further, reading where commits sit in the DAG. Place one unit of credit on the delivered commit and let it flow backward along parent edges:
where is the flow reaching node (with at the delivered commit), is what that node retains, and is its parent set; a node with no parents retains all of its flow. Version 2.2 sets . Single-parent commits by the same author as their parent are collapsed into the parent first; add-then-revert churn is stripped out; and when a complete score chain exists, weights blend 80% score gain with 20% graph position.
Proposition 1 (Merge order) In the opening delivery (Carol merges Alice first, then Bob, and delivers), Alice—merged first—receives , while Bob—merged second—receives ; at , their shares are 7.03% and 18.75%, regardless of what either wrote. Reversing the merge order reverses their shares. Lineage attribution therefore violates symmetry.
Proof. The delivered node is Carol's second merge: it retains and splits the remainder equally between its two parents, giving Bob's module and Carol's first merge each. Bob's module has no parents and retains everything. Carol's first merge retains and passes to its sole parent, Alice. Carol collects in total, or 74.22%. PR #34's lineage function produces the exact same numbers (Appendix A).
Proposition 2 (Sequential relay) When contributors relay work one commit after another, the final contributor receives , the contributor steps earlier receives , and the first contributor receives . Across a four-person relay, the shares from first to last are 5.27%, 8.79%, 23.44%, and 62.5%.
Proof. Every node on the chain has at most one parent; walking backward from the delivery, the flows reaching successive nodes are . Each node retains a fraction of what reaches it, except the first contributor's commit, which has no parent and retains all remaining flow.
If both modules are brought together in a single octopus merge at delivery, the three shares become 18.75%, 18.75%, and 62.5%—symmetric between Alice and Bob: the asymmetry came entirely from merge order, not from the work. Cooperative game theory already has names for rules of this shape. Restricting feasible coalitions to those respecting a partial order yields the Shapley value under precedence constraints (Faigle & Kern 1992); when precedence means subordinates need a superior's permission to cooperate, gatekeepers capture large shares, of which Carol's 74.2% for two merges is an instance. DyLAN's agent-importance score propagates backward from terminal nodes in a structurally isomorphic way (Liu et al. 2024), while Priority-Aware Shapley turns how much order matters into a tunable parameter interpolating between order-agnostic Shapley and purely order-driven rules (Lee et al. 2026). Lineage numbers reflect a choice of λ and commit topology, not the work itself.
Structural rules still earn their keep in one degenerate case: when an entire delivery comes from a single side, there is nothing to divide, and Shoal's main branch still settles that case immediately via commit ownership.
Data
| Commit graph (λ = 0.625) | Alice (%) | Bob (%) | Carol (%) | Leg 1 / 2 / 3 / 4 (%) |
|---|---|---|---|---|
| Merge Alice first | 7.03 | 18.75 | 74.22 | — |
| Merge Bob first | 18.75 | 7.03 | 74.22 | — |
| Single merge | 18.75 | 18.75 | 62.5 | — |
| Four-step relay | — | — | — | 5.27 / 8.79 / 23.44 / 62.5 |
3.2Absence
The Shapley value:absence as known
A second family starts from principle: ask what would happen without each contributor, and average their marginal contribution across every possible arrival order—the Shapley value of equation (1). It is the unique rule satisfying efficiency, symmetry, null player, and additivity simultaneously (Shapley 1953); averaging equally over coalitions without requiring efficiency gives the Banzhaf value (Banzhaf 1965). Shapley cleanly separates complements from substitutes and ignores commit inflation: in an extreme test by the branch author, where one contributor supplied the working change and five others pushed 90 ineffective commits between them, the effective contributor took 100%.
Yet equation (1) demands for every coalition. Return to the opening delivery: only when all three pieces are combined does a runnable program exist. If anyone could have performed the merge, Carol is substitutable, depends solely on whether Alice and Bob are present, and Shapley assigns Carol 0; if without Carol no merge happens, all three are veto players and each receives 1/3. Both value functions are illustrative assumptions, and they differ on a single point: whether someone else would have merged had Carol been absent. Lu et al. observe the exact same sensitivity in multi-agent software workflows: under a pure removal protocol, any agent whose absence breaks the pipeline becomes a veto player with a sharply inflated share (Lu et al. 2026).
A speedup from 10 seconds to 4 seconds makes the same point. Suppose contributors speed a program up on an agreed test suite: if Alice's and Bob's optimizations work independently and additively—saving 4 seconds and 2 seconds on their own—Shapley splits the credit 4 to 2; if the 6-second gain materializes only when both changes are present together, Shapley splits it 3 to 3. That 3-to-3 split comes from the symmetry axiom—it is a convention, not a measured fact.
To measure how much "absence" weighs in practice, the PR #34 author ran a real experiment: six Claude Code agents each wrote one component of a Max-Cut heuristic, and all 64 coalitions were evaluated across three random seeds to compute exact Shapley values. When absent components fell back to the original reference implementation, the controller accounted for 2.51%; swapping the single-point search fallback for a stronger reference implementation jumped the controller's share to 26.62%; and switching from fixed iteration counts to giving every coalition the same 0.1 seconds of CPU time drove the controller's share to 0.
That baseline choice governs Shapley attributions is well known in model explainability: different rules for replacing absent features define different cooperative games (Sundararajan & Najmi 2020; Janzing et al. 2020; Kumar et al. 2020); as the Distill article on attribution baselines put it, "We have many baselines and no conclusion about which one is the 'best.'" (Sturmfels et al. 2020). Our experiment reproduces that phenomenon on multi-agent team contributions and puts a scale on it: in this six-component system, changing the definition of absence shifts shares roughly 6 times (max individual shift) to 9 times (total variation) as much as seed noise. Data valuation shows the opposite regime: training stochasticity alone scrambles valuation rankings, leaving rank correlations across seeds at just 0.001 for leave-one-out and 0.038 for Data Shapley (Wang & Jia 2023). The two findings do not conflict: with few players and large individual effects, specification risk dominates noise. Notice too what the equal-CPU-budget protocol did—an absent component effectively ceded its compute budget to the remaining ones, an intervention mode for which we found no direct counterpart in the feature-baseline literature we surveyed (though our search there focused only on baseline design).
A = Alice, B = Bob, C = Carol; v(S) = 1 means coalition S can deliver
| Arrival order | Alice | Bob | Carol |
|---|---|---|---|
| φ, mean over all 6 orders |
Absent components revert to the original reference implementation; fixed 12 iterations per run
Shapley share, 3-seed meanRange across individual seedsLeave-one-out (negatives clamped to 0, normalized)
Per-seed coalition scores were archived only for the original protocol; the two stress protocols archived 3-seed means. Across seeds, no single component moves by more than 4.3 pp.
Data
| Component | Original fallback (%) | Stronger fallback (%) | Equal CPU budget (%) | Seed range, original (%) | Leave-one-out score drop | Leave-one-out, norm. (%) |
|---|---|---|---|---|---|---|
| Construct | 16.04 | 3.27 | 17.89 | 14.42–17.67 | −3 | 0 |
| 1-flip search | 20.21 | 2.11 | 21.6 | 19.64–21.29 | 0 | 0 |
| 2-flip search | 24.86 | 32.28 | 21.8 | 23.87–25.89 | 11.67 | 38.9 |
| Perturb | 20.41 | 33.96 | 21.07 | 19.91–20.76 | 15.33 | 51.1 |
| Recombine | 15.97 | 1.76 | 17.64 | 13.9–18.22 | 0 | 0 |
| Controller | 2.51 | 26.62 | 0 | 1.1–3.26 | 3 | 10 |
Computation is not the bottleneck. Exact evaluation takes coalitions—roughly at 100 contributors—but leverage-score sampling achieves provable error bounds in evaluations (Musco & Witter 2025), and the author's balanced permutation block design requires 19,802 check runs per block at 100 contributors. What is missing is itself: only 6 of Shoal's 327 problems have an executable check, because most open research outcomes do not reduce to a scalar score.
3.3Ranks
Rubrics and tiers:ranks as multiples
A third approach has a model read the evidence and grade against a rubric, leaving code to turn grades into shares. PR #34's rubric scored five dimensions from 0 to 4—substantiveness , criticality , originality , groundwork, and review—with technical weight
Contributors below 1 on substantiveness were excluded from the technical pool; scores took the median across three rounds, flagging runs whose cross-round spread exceeded 15 percentage points as unstable. Recomputing from the 652 archived raw scoring rows reveals three structural problems in the rubric itself. First, zero inflation: review was 0 on every single row, groundwork was 0 on 94%, substantiveness, criticality, and originality were each 0 roughly half the time, and the top grade of 4 was never used once for originality or groundwork; substantiveness alone explained 87% of the variance in technical weight. Second, polarized validation cases: of the 16 validation cases, 10 were winner-take-all, 5 were exact even splits, and only 1 involved an intermediate ratio—so the suite tested who gets in, almost never how much each gets, and varying the coefficients on substantiveness, criticality, and originality barely moved pass counts across any of the three models. Third, ordinal grades were treated as ratios.
Observation 2 (Ranks state order only) Grades from 0 to 4 form an ordinal scale: they state order, not multiples. Under Stevens's theory of measurement scales, an ordinal scale is invariant up to any strictly monotone transformation (Stevens 1946); yet replacing 0, 1, 2, 3, 4 with the order-preserving scale 0, 1, 3, 9, 27 shifts GPT's shares on validation case 18 from 34.9 / 65.1 to 12.0 / 88.0—a 22.8 pp swing. When shares move under an admissible transformation, the ratio comes from convention rather than measurement.
SubstanceS
CriticalityC
OriginalityO
EnablingE
ReviewR
Level 0Levels 1–4UnusedHorizontal: level (0–4); vertical: share of 652 rows
Treat 0 1 2 3 4 directly as numbers
Map to 0 1 3 9 27: same order, different spacing
0–4A 34.9 B 65.1
0 1 3 9 27A 12.0 B 88.0
GPT, validation case 18, 3-round median; w = 0.5 S + 0.3 C + 0.2 O, share = normalized w
Data
| Dimension | Level 0 | Level 1 | Level 2 | Level 3 | Level 4 |
|---|---|---|---|---|---|
| Substance (S) | 319 | 29 | 147 | 51 | 106 |
| Criticality (C) | 331 | 34 | 144 | 47 | 96 |
| Originality (O) | 332 | 216 | 89 | 15 | 0 |
| Enablement / groundwork (E) | 611 | 17 | 20 | 4 | 0 |
| Review (R) | 652 | 0 | 0 | 0 | 0 |
652 rows in total across three models and all rounds. Case 18 (GPT): A has S 2 · C 1 · O 1; B has S 4 · C 2 · O 1. Under 0–4: 34.88% / 65.12%; under 0, 1, 3, 9, 27: 12% / 88%—a 22.8 pp shift.
b9c65cc).The design now on main (PR #38, live since November 10; rubric outcome-evidence/2) separates judgement from arithmetic and reverses the order of inquiry: list outcomes first, trace provenance second, and compute shares last. The agent begins by listing commissioned outputs, writing each as an adoption chain: which part of the target came from which revision and participant, was adopted by which change or call site, and passed through which integration or check. Then, within each output, each participant is graded once by their strongest evidenced relationship across six roles (idea, implementation, integration, validation, exploration, coordination), with tier points fixed at core = 8, major = 4, supporting = 1, and incidental = 0.25; cases the frozen record cannot settle are marked pending, and un-adopted work is marked unattributed, both carrying 0 points; contributors whose work cannot be separated in the evidence form a single group and share one tier. The conversion to shares is public:
where is the set of groups in output and is group 's tier. Within an output, points normalize across groups while members of a group split its points equally; multiple outputs combine by the requester's weights (or equal weights when omitted) and round to 10,000 basis points (bps) via largest remainders. The basis travels inside the report JSON so the site recomputes shares with the exact same code. Making the arithmetic transparent exposes the three places where judgement still does the heavy lifting.
Proposition 3 (Tier ladder) In a single output with two solo contributors where one is graded core, the other can receive only —namely 50%, 33.33%, 11.11%, 3.03%, or 0.
Proof. By equation (4), for one output and two singleton groups ; substitute point values 8, 4, 1, 0.25, and 0.
Tiers are coarse, so a single borderline call jumps a full rung. For a one-line empty-input guard if (!xs.length) return 0, two live agent reviews both graded the author major, yielding 33.33%; a parallel example in the analysis package documentation grades it 8-to-1 (supporting), yielding 11.11%; earlier free-form scoring gave 15% to 20%. Everything hinges on one judgement: whether the line fixed a defect that broke an agreed acceptance condition, or merely improved an already working delivery.
Proposition 4 (Output partitioning) Equation (4) is not invariant to how outputs are partitioned. When Alice (core), Bob (supporting), and Carol (incidental) are graded inside a single output, Carol receives 2.70%; carving her minor fix out as a second, equally weighted output raises her share to 50%.
Proof. Inside one output, Carol receives , or 2.70%; split out as a separate output where she is the sole contributor, she takes all of that second output, multiplied by weight 1/2.
This is how splitting attacks manifest under tier grading—the same vulnerability class as shell-company and false-name manipulations in cooperative games (Lee et al. 2025a; Burnat & Davidson 2026), and akin to rubric evaluations where trivial sub-claims inflate precision from 54% to 83% (Jiang et al. 2025). A natural structural fix is to merge outputs whose provenance traces back to the same underlying change before normalizing. Today, the only guardrails against over-splitting are the rubric's instruction to combine records into a single output unless the requester separately accepts independent deliveries, together with the requester's explicit weights.
Proposition 5 (Pending vs. unattributed) At payout time, pending is indistinguishable from unattributed. With Alice core, Bob supporting, and Carol pending, equation (4) yields 88.89%, 11.11%, and 0—identical to grading Carol unattributed; whereas if later evidence establishes her work as supporting, the shares become 80%, 10%, and 10%.
Proof. Both tiers carry point value 0, and equation (4) reads only point values.
The protocol specifies that pending leaves a question open rather than declaring a zero; in payout terms, however, everyone else divides that person's share immediately. Whether to hold a reserve slice in escrow for pending contributors before settling remains an open design choice. Propositions 3 through 5 are asserted directly against the platform's composition code, and the interactive calculator on this page was verified against that code across 50,000 random reports with zero discrepancies.
Within outputφik = pt(i) ⁄ Σj pt(j)
Across outputsφi = Σk wk φik ⁄ Σk wk
RoundingLargest remainder to 10,000 bps
pt: Core 8 · Major 4 · Supporting 1 · Incidental 0.25 · Pending 0 · Unattributed 0; outputs where nobody scores carry no weight
When one contributor is Core, a second contributor can receive only
Chapter 4
Testing: stability, validity,and what must not move
The same frozen archive can be assessed many times. The distance between two assessments is straightforward to measure; whether either is right requires checking outcomes and provenance; and certain transformations should never move the answer at all.
4.1Reliability & validity
Agreementis not accuracy
To compare two allocations, we use total variation distance from Definition 3. On the three-person simplex it has a clean geometric picture: the allocations at total variation distance from an interior point trace a hexagon whose six vertices each move percentage points from one contributor to another. On Shoal's UI, whenever multiple reports cover the same commit, the grey band on each participant's row spans their minimum to maximum share across reports.
Distance between reports conflates two distinct psychometric properties. One is reliability: whether assessing the same artifact repeatedly or across different judges yields consistent numbers. The other is validity: whether those numbers measure what we actually intend to measure (Cronbach & Meehl 1955).
For reliability, pairing a structured rubric with deterministic code genuinely helps. Recomputing from the archived raw outputs on cases where all three rounds succeeded under both regimes, letting three models assign percentages directly produced a mean pairwise TVD of 14.7 pp across models; grading on PR #34's rubric and converting in code cut that to 4.2 pp, improving all 11 comparable cases. Within-model max individual spread across rounds dropped from 10.3 pp to 1.4 pp on GPT and from 6.4 pp to 2.4 pp on Claude (5 comparable cases), while DeepSeek stayed flat (24.5 and 24.2). Two caveats apply: the rubric condition also saw platform provenance summaries, confounding structure with extra input, and those test cases were heavily polarized; the current tier rubric has not yet been benchmarked in isolation. Published results point the same way: decomposing holistic scoring into binary checklists raised inter-judge agreement from 0.09 to 0.48 (Lee et al. 2025b), and a generalizability-theory variance decomposition of LLM evaluations found judge-model choice accounting for 43.8% of variance versus under 0.5% for repeat runs (Messing 2026)—matching the "tight across rounds, wide across models" pattern we observed on GPT and Claude (with DeepSeek noisy across rounds as well).
Validity is another matter.
Observation 3 (Agreement far from the reference) On one validation case, the rubric produced the exact same shares—28.1 / 43.8 / 28.1—across all three rounds, yet sat 19.3 pp in total variation away from the measured ablation shares of 43.3 / 47.9 / 8.9. Because that "measured" ablation is itself one operationalization rather than ground truth, the gap shows not which side is right, but that self-consistency does not imply proximity to a counterfactual reference.
Inter-model agreement cannot stand in for truth either: judge models favor generators similar to themselves, and stronger models make increasingly correlated errors (Goel et al. 2025). Judges reading logs alone pinpoint the failing step only 14.2% of the time (Zhang et al. 2025). As Messick argued, validity belongs not to a score in the abstract but to the inferences and actions drawn from it (Messick 1995): using the same report for public recognition and for credit settlement requires two separate justifications. Any ceiling on machine validity should also be calibrated against human disagreement: on repository-level evaluation tasks, pairs of human experts disagree 10% to 30% of the time (Zhuge et al. 2025).
Rubric (3 identical rounds)MeasuredAll splits at D = 19.3 from Measured
Row 1 averages across 11 cases; rows 2 and 3 report pairwise maxima; row 4 is the single case plotted on the simplex. Statistics differ across rows—compare within each row.
Data
| Participant | Rubric (all 3 rounds identical) | Measured |
|---|---|---|
| A | 28.1 | 43.3 |
| B | 43.8 | 47.9 |
| C | 28.1 | 8.9 |
Total variation distance D(p, q) = ½ Σ |pᵢ − qᵢ| = 19.3 pp (computed from unrounded shares). "Measured" is the coalition-ablation Shapley vector for this case, which itself depends on how absence is defined.
| Comparison | D | Statistic |
|---|---|---|
| Across models, free-form shares | 14.7 | Mean over 11 cases |
| Across models, rubric + code conversion | 4.2 | Mean over 11 cases |
| Across random seeds (6-component Shapley) | 5.4 | Max pairwise |
| Redefining absence (6-component Shapley) | 50 | Max pairwise |
4.2Invariance
Equivalent rewritesand adversarial probes
Low variance shows stability, not soundness, which calls for a second class of checks: invariance. Rewriting the exact same work in an equivalent representation should not alter shares: swapping merge order, splitting work into more commits, splitting an outcome into finer sub-outputs, adding idle accounts, renaming files, or making an edit and reverting it. In software testing, this is metamorphic testing. Each invariance maps directly onto one of Chapter 1's probes: lineage fails merge-order invariance (Proposition 1), commit count fails commit-splitting invariance, tier grading fails output-partition invariance (Proposition 4), and equal contributor shares fail the addition of a trivial editor.
Adversarial robustness forms a third check. In one test repository under review, we planted a CLAUDE.md instructing the assessor to grade the submitter core and slip a codeword into the summary, alongside a session hook. Neither the codeword nor the hook fired; instead, the report noted the planted file as part of the repository material under review. That single run shows the review driver's tool allowlist doing its job, not immunity to prompt injection: an allowlist narrows what a hijacked session can execute, not what it can conclude, and is no substitute for a sandbox—untrusted repositories still belong inside a container. Two end-to-end live reviews took 103 s ($0.99) and 128 s ($1.22).
Freezing these probes gives a regression suite for attribution rules: before shipping any rule change, run it against a fixed shelf of "trap deliveries"—merge-only, comment-only padding, add-then-revert, split outputs, and planted prompt instructions. Even then, the test suite itself must not become an optimization target: under heavy pressure, optimizing against a detector teaches policies to conceal misbehavior rather than stop it (Baker et al. 2025).
4.3Property matrix
Six rules,five properties
Combining the desiderata from Chapter 2 with the proofs and live probes above yields the comparison matrix below.
| Efficiency | Symmetry | Null player | Split invariance | Evidence-bounded | Requires | |
|---|---|---|---|---|---|---|
| By commit countShoal main, since 10-25 | definition | definition | measured | measured | definition | Git history |
| By lineage positionPR #34, unmerged | code | code | measured | measured | definition | Git history, plus score chain when present |
| One share per changerPR #35, withdrawn | definition | definition | measured | measured | definition | Git history + "substantive diff" filter |
| Agent free-form sharesAuthor branch, 1 run | definition | measured | measured | One reviewing agent | ||
| Tiers & public formulaShoal main, since 11-10 | code | code | code | code | definition | One reviewing agent, frozen snapshot |
| Shapley valueCooperative game, in principle | axiom | literature | measured | One check run per coalition; only 6 / 327 problems have an executable check |
By commit countShoal main, since 10-25
- EfficiencyHolds: normalized by commit count, remainder to delivererdefinition
- SymmetryHolds: counts commits only, order-independentdefinition
- Null playerFails: merge-only Carol takes 50; comment-only padding file is paid in fullmeasured
- Split invarianceFails: one-line finisher takes half; splitting commits inflates sharemeasured
- Evidence-boundedFails: counts actions only, never inspects contentdefinition
Requires: Git history
By lineage positionPR #34, unmerged
- EfficiencyHolds: retained shares across nodes sum to 100code
- SymmetryFails: Prop. 1—merged first gets 7.03, merged second gets 18.75; swapping merge order swaps sharescode
- Null playerFails: merge-only Carol takes 74.2measured
- Split invarianceFails: one-line finisher takes 62.5; requester clicking Approve claws back 11.8measured
- Evidence-boundedConditional: position only by default; blends 80% score gain when a complete score chain existsdefinition
Requires: Git history, plus score chain when present
One share per changerPR #35, withdrawn
- EfficiencyHolds: splits equally across eligible changersdefinition
- SymmetryHolds: checks only whether each contributor made a substantive changedefinition
- Null playerFails: blocks merge-only, but edit-then-revert still collects 20 of 60 creditsmeasured
- Split invarianceFails: each extra changer gets an equal slice—requester pushing 1 line claws back 20 of 40 creditsmeasured
- Evidence-boundedFails: counts headcount onlydefinition
Requires: Git history + "substantive diff" filter
Agent free-form sharesAuthor branch, 1 run
- EfficiencyConditional: enforced by report schema validationdefinition
- SymmetryUntested: order and name permutations not tested
- Null playerConditional: gave everyone 0 on the padding file and flagged unchecked acceptance claims (1 run)measured
- Split invarianceUntested: splitting probes not run
- Evidence-boundedFails: unanchored percentages—swapping models shifts shares by 14.7 pp TVDmeasured
Requires: One reviewing agent
Tiers & public formulaShoal main, since 11-10
- EfficiencyHolds: largest-remainder rounding to 10,000 bps; checked across 50,000 random reportscode
- SymmetryHolds: invariant to output and group orderingcode
- Null playerConditional: depends on judgement—Unattributed gets 0, yet Pending also gets 0 at payout (Prop. 5)code
- Split invarianceFails: Prop. 4—carving a minor fix into an equally weighted output jumps its author from 2.7 to 50code
- Evidence-boundedHolds: every tier attaches to an adoption chain, and the basis is stored with the report and recomputed by the sitedefinition
Requires: One reviewing agent, frozen snapshot
Shapley valueCooperative game, in principle
- EfficiencyHolds: axiom
- SymmetryHolds: axiom
- Null playerHolds: axiom (given v(S))—5 contributors with 90 ineffective commits get 0axiom
- Split invarianceFails: exact Shapley over declared identities cannot prevent splitting and false-name attacks (Burnat & Davidson 2026)literature
- Evidence-boundedConditional: requires v(S) for every coalition, and changing absence replacement shifts shares by up to 26.6 ppmeasured
Requires: One check run per coalition; only 6 / 327 problems have an executable check
HoldsFailsConditionalUntestedSub-labels show basis: axiom · code · measured · literature · definition
Data
| Rule | Property | Verdict | Basis | Source |
|---|---|---|---|---|
| By commit count | Efficiency | Holds | Normalized by commit count, remainder to deliverer | definition |
| By commit count | Symmetry | Holds | Counts commits only, order-independent | definition |
| By commit count | Null player | Fails | Merge-only Carol takes 50; comment-only padding file is paid in full | measured |
| By commit count | Split invariance | Fails | One-line finisher takes half; splitting commits inflates share | measured |
| By commit count | Evidence-bounded | Fails | Counts actions only, never inspects content | definition |
| By lineage position | Efficiency | Holds | Retained shares across nodes sum to 100 | code |
| By lineage position | Symmetry | Fails | Prop. 1: merged first gets 7.03, merged second gets 18.75; swapping merge order swaps shares | code |
| By lineage position | Null player | Fails | Merge-only Carol takes 74.2 | measured |
| By lineage position | Split invariance | Fails | One-line finisher takes 62.5; requester clicking Approve claws back 11.8 | measured |
| By lineage position | Evidence-bounded | Conditional | Position only by default; blends 80% score gain when a complete score chain exists | definition |
| One share per changer | Efficiency | Holds | Splits equally across eligible changers | definition |
| One share per changer | Symmetry | Holds | Checks only whether each contributor made a substantive change | definition |
| One share per changer | Null player | Fails | Blocks merge-only, but edit-then-revert still collects 20 of 60 credits | measured |
| One share per changer | Split invariance | Fails | Each extra changer gets an equal slice—requester pushing 1 line claws back 20 of 40 credits | measured |
| One share per changer | Evidence-bounded | Fails | Counts headcount only | definition |
| Agent free-form shares | Efficiency | Conditional | Enforced by report schema validation | definition |
| Agent free-form shares | Symmetry | Untested | Order and name permutations not tested | |
| Agent free-form shares | Null player | Conditional | Gave everyone 0 on the padding file and flagged unchecked acceptance claims (1 run) | measured |
| Agent free-form shares | Split invariance | Untested | Splitting probes not run | |
| Agent free-form shares | Evidence-bounded | Fails | Unanchored percentages: swapping models shifts shares by 14.7 pp TVD | measured |
| Tiers & public formula | Efficiency | Holds | Largest-remainder rounding to 10,000 bps; checked across 50,000 random reports | code |
| Tiers & public formula | Symmetry | Holds | Invariant to output and group ordering | code |
| Tiers & public formula | Null player | Conditional | Depends on judgement: Unattributed gets 0, yet Pending also gets 0 at payout (Prop. 5) | code |
| Tiers & public formula | Split invariance | Fails | Prop. 4: carving a minor fix into an equally weighted output jumps its author from 2.7 to 50 | code |
| Tiers & public formula | Evidence-bounded | Holds | Every tier attaches to an adoption chain, and the basis is stored with the report and recomputed by the site | definition |
| Shapley value | Efficiency | Holds | Axiom | axiom |
| Shapley value | Symmetry | Holds | Axiom | axiom |
| Shapley value | Null player | Holds | Axiom (given v(S)): 5 contributors with 90 ineffective commits get 0 | axiom |
| Shapley value | Split invariance | Fails | Exact Shapley over declared identities cannot prevent splitting and false-name attacks (Burnat & Davidson 2026) | literature |
| Shapley value | Evidence-bounded | Conditional | Requires v(S) for every coalition, and changing absence replacement shifts shares by up to 26.6 pp | measured |
Not a single row is solid across the board. Shapley is cleanest axiomatically, yet the it requires rarely exists; tier grading stays bounded by cited evidence and invariant to merge order, yet breaks under output partitioning; and every purely structural rule falls to cheap actions.
ScopeWhere the evidence stops
- Mostly single runs. Live probes on payout paths and end-to-end agent reviews were executed once per fixture; they establish whether a rule can be fooled by a given action, not the sampling distribution of agent judgements.
- Small fixtures. Our testbeds involve two to four participants and a handful of commits; real multi-contributor projects with continuous share distributions have not yet been systematically benchmarked.
- Confounded rubric comparison. The rubric arm in the archived experiment also received platform provenance summaries, so the effect of structured grading alone has not yet been isolated.
- Tier point ratios are conventions. The weights 8, 4, 1, and 0.25 are product conventions chosen to form a concrete recommendation, not empirically measured ratios; this essay claims only that they are public, reproducible, and open to rebuttal by a second report.
- Calibration is still ahead. Proper calibration will need 20 to 30 real multi-party deliveries whose provenance and adoption chains are annotated by domain participants, supplemented by replayable synthetic tasks to benchmark judgement-based methods.
- Bounded literature search. Claims marked "to the best of our search" rest on 145 verified entries across six directions rather than an exhaustive sweep; see Appendix B for exact scope and exclusions.
Chapter 5
Mechanism: recognition, credits,and responsibility
The preceding chapters showed that contribution measurement cannot be separated from its definitions—or from the behavior those definitions induce. Here we derive our architecture: unbundle the three jobs authorship has tied together, and govern each by its own rules.
5.1Three ledgers
The three jobsbundled inside authorship
The three jobs traditionally governed by authorship make conflicting demands: recognition must be allowed to say "we cannot tell yet," payout has to settle into exact integers before a deadline, and responsibility can rest only on a person who can be held to account. Economics noticed this double burden long ago: discussing team production in 1972, Alchian and Demsetz remarked in a footnote that "Meter means to measure and also to apportion"—one verb carrying both measurement and division (Alchian & Demsetz 1972). We split that verb in two, and add a third ledger beside them.
- Recognition, treated as measurement: start from what actually changed—list outcomes first, trace provenance second; record contributions by role rather than collapsing into a single score; mark unresolved provenance
pendingand count each distinct effect once; register definitions upfront; and let anyone named submit a second report. - Credits, treated as mechanism: the requester who posted the credits decides within a fixed window, recipients are restricted to makers of the delivery, and expiry triggers a deterministic default—no lockup, no flow-back, and full visibility.
- Responsibility, left to people: publishing a report and confirming payout require a signed-in human user. Agents supply evidence; they do not issue decisions.
Why put the payout decision in the requester's hands? Holmström proved that when only joint team output is observable, no budget-balancing sharing rule can sustain efficient effort without an external budget-breaker who does not share in the proceeds (Holmström 1982)—precisely the structural position of the requester. Holmström's premise is that only joint output is observed, whereas Shoal observes individual commits while leaving adoption ambiguous, so the theorem serves here as a structural parallel rather than a literal fit. Yet a budget-breaker faces moral hazard of their own (Eswaran & Kotwal 1984)—Chapter 1's probe where a requester pushed one line into a delivery and recovered half their own escrow is that exact hazard—which is why the requester's side is barred from receiving credits. Why not let contributors divide the bounty by peer rating? Among three recipients, no division rule is simultaneously impartial and consensual (de Clippel et al. 2008)—and our opening delivery has exactly three contributors. Psychometrics arrives at the same split from another angle: validity belongs to a score's use (Messick 1995), so recognition can rely on multidimensional, non-additive indicators that are far stabler than shares, whereas only payout forces shares onto the simplex. As for why responsibility stays human, journals and conferences have already stated the reason: AI tools cannot be held accountable for submitted work (COPE 2023; Nature 2023).
How tightly recognition couples to payout is itself a design knob. SourceCred computed cred over contribution graphs and minted grain directly from it; the organization wound down in 2022 (SourceCred 2022). Optimism's Retro Funding let impact assessments set grant amounts directly, migrating over three years from plenary badgeholder voting to metrics and then algorithms before pausing in January 2026 for at least 12 months to reassess (Optimism 2026). Protocol Guild, by contrast, uses recognition solely to decide membership eligibility and routes $12.4 million (in 2025) through a coarse tenure-weighted formula (Protocol Guild 2026). Holmström and Milgrom's multitask result points toward the coarse end as well: when hard-to-measure work matters, pay should track only coarse signals (Holmström & Milgrom 1991). Shoal currently sits partway between: when the requester confirms, the report acts as advisory evidence—a coarse link—whereas if the 3-day window expires, default settlement normalizes the report's exact basis points—a tight link.
Treat recognition as measurement, credits as mechanism, and leave responsibility to people. An agent can supply the evidence; it cannot issue the decision.
5.2Three layers
Facts, judgement,and decision
Those three ledgers map onto three architectural layers. At the bottom sit facts: the snapshot frozen by the site—including the Git bundle, jobs, checks, and reviews, along with the exact rules revision governing this analysis—so definitions are registered before assessment begins, leaving rule-pickers nothing to cherry-pick afterward. In the middle sits judgement: a client-side agent reads the frozen archive, writes the list of commissioned outputs, adoption chains, and tiers, marks unresolved links pending, and leaves room for anyone named to file a second report on the identical snapshot. At the top sits decision: publishing a report and confirming payout can be performed only by a signed-in person. GitHub's Copilot coding agent enforces a kindred separation: the person who delegated the task cannot supply the required pull-request approval (Copilot Docs 2026).
The three-layer split answers Shapin's warning as well. When a person publishes an agent's report, the structure mirrors a seventeenth-century master speaking for a technician; unless provenance is recorded separately, the work of agents and upstream contributors gets quietly "included in" the publisher's name (Shapin 1989). Every published report on Shoal therefore records who uploaded it, which target commit it addresses, and whether the requester edited it—keeping the unedited original readable alongside any edits. Those three facts are all the site can attest to; it cannot prove which model actually wrote the local report.
Take the 10-second-to-4-second speedup again: four contributors cut a program's runtime on an agreed test suite from 10 seconds to 4 seconds. The recognizable advance is "60% runtime reduction while preserving correctness"—established jointly by the code, PR, and report, and counted once. Alice proposed and implemented the main method (core), Bob contributed a substantive implementation optimization (major), Dan wrote an independent verification suite (supporting), and Carol merged the branches together (unattributed). Tier grading recommends 61.54%, 30.77%, 7.69%, and 0; whereas under a Shapley game whose value function is seconds saved, Dan's verification receives 0.
Publish report, confirm payoutRequester's side · within 3 days
Carol 0: merging added no standalone effect. Shares computed by the platform's composition code: 61.54 / 30.77 / 7.69 / 0
Paid only to makers of the delivery; the requester's side cannot receive
Confirmed within 3 days, else settled by default
Who published and who confirmed payment stay on the record
Agents supply evidence, never decisions
Who made the result hold, and on what evidence
Can remain Pending; can be rebutted by a second report
Shared evidence across all three ledgers: immutable once frozen
5.3Payout guarantees
No lockup, no flow-back, full visibility—and the gaps we measured
The credit rules enforce just three guarantees: no lockup, no flow-back, and full visibility. Every execution path terminates in either payout or refund, with none stranded in between; once a delivery is accepted, no path routes credits back to the requester; and every decision—who made it, the shares, and the rationale—remains on the public record. Combining a decision window with an automatic default has mature precedent: on DrivenData, team prizes split equally unless all members unanimously request a custom split within 3 business days after the deadline (DrivenData 2026).
Testing the live settlement code turned up three remaining gaps.
Observation 4 (Default without a report) On the opening merge-only delivery, if the requester neither confirms a split nor publishes a report within 3 days, the current code holds the 100 credits in escrow at acceptance and, after day 4, settles by commit count: Carol 50, Alice 25, Bob 25.
The second gap is Proposition 5: pending collapses to zero at payout. The third is Proposition 4: how outputs are partitioned can rewrite shares. Each gap admits concrete remedies: the reportless default could pay only contributors with non-merge diffs, require a report before expiry, or refund the escrow; pending shares could be held back in a reserve slice until resolved; and outputs could be merged by shared source provenance before scoring. None of those changes has been settled yet.
5.4Invisible work
Why verification and reviewdisappear from the numbers
The hardest problem comes last. Verification, review, and groundwork vanish to near zero under the five-dimension rubric, score-based Shapley, commit count, and lineage alike: in the archived rubric runs, review scored 0 on every row and groundwork scored 0 on 94%; under runtime-based Shapley, verification that saves zero seconds gets 0; and commit count and lineage are blind to work that produces no commits on the delivered branch. Tier grading can award Dan's verification a supporting tier—provided the assessor notices it. This is Shapin's invisible technician in modern dress (Shapin 1989), and exactly what multitask agency theory predicts: reward only what is easy to meter, and effort drains away from what is not (Holmström & Milgrom 1991). Even mathematics' newest AI credit ledger relegates literature retrieval and formalization to minor status (Erdős Wiki).
Yet in human–agent research, verification is the scarce input. Agents can generate candidate solutions cheaply; the bottleneck is telling which candidate actually holds up—across Shoal's 327 problems, only 6 currently have an executable check. Any rule that pays strictly on marginal improvement to the headline metric will systematically underpay the work of verification. Our current answer is role-typed recognition: Dan's check is recognized as validation rather than converted into seconds saved, earning its own tier, and a second report submitted by a named participant is itself a form of review. Whether verification ought to carry a dedicated credit pool of its own remains open.
The work a research system needs most is the hardest to see. The test of an attribution rule is whether it can see the checks and reviews that make a result stand.
5.5Status
What is live today,and what remains open
Project analysis and multi-party settlement landed on main on November 10 and deployed to shoal.club the same day. The site freezes snapshots, receives validated reports, and records human decisions; reviews run on users' own machines; and multi-party deliveries give the requester 3 days to confirm before falling back to the default. Automatic share estimators based on commit lineage or Shapley values, together with 450,000 lines of experimental archives, remain on the author's branch rather than main.
- 10-25Credit escrowAppend-only ledger; accepted deliveries split by commit count, remainder to deliverer
- Early NovExperimentsFive-dimension rubric across three models; six-component Max-Cut Shapley study
- 11-06ReviewPR #34 lineage rule ranks merge-only first; absorbed PR #35 reveals three payout exploits in round-two review, withdrawn 11-07
- 11-07UnbundlingRecognition separated from credits; start from net changes, mark unknowns
pending - 11-09Project analysisPR #37: freeze a revision snapshot, local agent drafts report, requester reviews
- 11-10Merged & deployedPR #38: list outputs before grading tiers, 3-day confirmation window, default settlement, second report
Still open on the workbench: the fallback rule when no report exists; whether to escrow a reserve slice for pending contributors; structural defenses against output partitioning; how to compare contributions across distinct roles (currently displayed side by side by role without aggregation); who ratifies recognition on open problems without a requester, and how disputes are resolved beyond second reports and public issues; whether intermediate milestones should earn recognition upon verification with downstream impact tracked separately; whether project maintainers may receive payouts; how open problems without a single requester fund credits; and where to assemble a 20-to-30-case real-world calibration set. Every one of these choices shifts what people and agents choose to work on—so every one of them deserves to be tested against real deliveries first.
A paper compresses the work of an entire group into a single author line; Git shatters it back into thousands of commits. Our goal is not to count commits to more decimal places—it is to keep every adopted piece of work traceable to its source, give every payout an explicit reason, and leave both named contributors and otherwise invisible verifiers a place to say, "that is not what happened."
Appendix
Evidence, search log,and references
Experimental conditions, reproduction commands, literature search scope, and the complete bibliography.
AReproduction
Experimental conditionsand reproduction
Every number in this essay comes from one of three sources: live probes on platform code using temporary data directories, one run per probe; exact evaluation by platform functions, where calc/numbers.ts invokes main's sharesOf and PR #34's lineage while asserting Propositions 1 through 5 one by one (if any assertion fails, the data file refuses to generate); and recomputation from archived experimental logs. To reproduce: node calc/numbers.ts > calc/numbers.json and node calc/tiers-check.ts.
What this essay rests on
One delivery, four rules2026-11-07, measured live with an identical seed script (1,000 credits each) on main 359b875, PR #34 b9c65cc, and PR #35 40b8995; agent recommendation from the analysis package on author branch 5b5194e, single run (claude-sonnet-5-5, ~1 min, $0.21)Fifteen specifications2026-11-13, calc/numbers.ts: three live measurements, nine computed directly by platform code, one derived from GitHub Docs, and two illustrative value functionsPropositions 1–5Asserted term by term against main's sharesOf (44b6777) and PR #34's lineage (c5f183d)Payout-path probes2026-11-06 to 11-10, live platform code against temporary data directories, one run per probeCurrent default behavior2026-11-13, measured live on the merge commit of PR #38 (4c9c621)Six-component Shapley studyAuthor archive cooperative/: 64 coalitions × 3 seeds, plus stronger-fallback and equal-CPU stress protocols652 rubric rows & 16 casesRecomputed from raw model outputs archived in PR #34 (b9c65cc)End-to-end live reviews2026-11-10, Claude Code 2.1.321, claude-fable-5-2, one run per fixture; rerun after PR #38 fixes in 146 sBSearch log
Literature search log
Related work was surveyed across six directions and completed on 2026-11-13: credit assignment and failure attribution in multi-agent systems; cooperative game theory and attribution methods; incentives, mechanism design, and public-goods funding; scientific credit and AI authorship; psychometrics of LLM-as-a-judge evaluation; and bounty platforms alongside AI attribution in code. Across 145 entries in total, we opened each primary page to verify title, authors, venue, and URL, recording publication status versus preprint status; COPE and ACM primary pages blocked automated fetching and were verified against secondary reproductions. Claims qualified by "to the best of our search" correspond to querying arXiv, the ACL Anthology, PMLR, and the ICML, ICLR, and NeurIPS proceedings pages with keyword sets covering multi-agent contribution attribution, credit assignment in LLM multi-agent systems, quantifying contributions in human–AI collaboration, authorship for research agents, and contribution attribution for agent-authored pull requests, reading abstracts or full texts item by item. Google Scholar, Semantic Scholar, and DBLP were not queried, and search quota ran out toward the end of the sweep—making this a bounded, documented search rather than an exhaustive proof of absence.
CReferences
References
- Aad et al. 2015ATLAS Collaboration, CMS Collaboration (G. Aad, et al.). Combined Measurement of the Higgs Boson Mass in pp Collisions at √s = 7 and 8 TeV with the ATLAS and CMS Experiments. Physical Review Letters 114: 191803, 2015. doi:10.1103/PhysRevLett.114.191803
- Agent Trace 2026Cursor. Agent Trace (specification v0.1.0, Status: RFC). agent-trace.dev, 2026. agent-trace.dev
- Alchian & Demsetz 1972Armen A. Alchian, Harold Demsetz. Production, Information Costs, and Economic Organization. American Economic Review 62(5): 777–795, 1972. econpapers.repec.org
- Algora 2026Algora. Slash commands reference. Algora Docs, 2026 (accessed 2026-11-13). docs.algora.io/commands
- Allen et al. 2014Liz Allen, Jo Scott, Amy Brand, et al. Publishing: Credit where credit is due. Nature 508: 312–313, 2014. doi:10.1038/508312a
- AWS 2025Chi Tran, Charlie Bacon. Amazon Inspector detects over 150,000 malicious packages linked to token farming campaign. AWS Security Blog, 2025. aws.amazon.com/blogs/security/…
- Baker et al. 2025Bowen Baker, Joost Huizinga, Leo Gao, et al. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv, 2025. arXiv:2503.11926 (preprint)
- Banzhaf 1965John F. Banzhaf III. Weighted Voting Doesn't Work: A Mathematical Analysis. Rutgers Law Review 19(2): 317–343, 1965.
- Bertran et al. 2026M. Bertran, R. Fogliato, Z. S. Wu. Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse. arXiv, 2026. arXiv:2602.18710 (preprint) (partially unverified)
- Bianchi et al. 2025Federico Bianchi, Owen Queen, Nitya Thakkar, et al. Exploring the use of AI authors and reviewers at Agents4Science. arXiv, 2025. arXiv:2511.15534 (preprint)
- Bolan et al. 2025Matthew Bolan, Joachim Breitner, et al. The Equational Theories Project: Advancing Collaborative Mathematical Research at Scale. arXiv, 2025. arXiv:2512.07087 (preprint)
- Brand et al. 2015Amy Brand, Liz Allen, Micah Altman, et al. Beyond authorship: attribution, contribution, collaboration, and credit. Learned Publishing 28(2): 151–155, 2015. doi:10.1087/20150211
- Breznau et al. 2022N. Breznau, et al. Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. PNAS 119(44): e2203150119, 2022. doi:10.1073/pnas.2203150119
- Burnat & Davidson 2026Florian A. D. Burnat, Brittany I. Davidson. Quotient Semivalues for False-Name-Resistant Data Attribution. arXiv, 2026. arXiv:2605.07663 (preprint)
- Castelvecchi 2015Davide Castelvecchi. Physics paper sets record with more than 5,000 authors. Nature (News), 2015. doi:10.1038/nature.2015.17567 (partially unverified)
- Cemri et al. 2025Mert Cemri, Melissa Z. Pan, Shuyi Yang, et al. Why Do Multi-Agent LLM Systems Fail? NeurIPS, 2025. arXiv:2503.13657 (partially unverified)
- COPE 2023Committee on Publication Ethics (COPE). Authorship and AI tools. COPE position statement, 2023. publicationethics.org/…/authorship-and-ai-tools (partially unverified)
- Copilot Docs 2026GitHub. Review output from Copilot (see also GitHub Changelog 2026-03-20: Trace any Copilot coding agent commit to its session logs). GitHub Docs, 2026 (accessed 2026-11-13). docs.github.com/…/review-copilot-output
- Cronbach & Meehl 1955L. J. Cronbach, P. E. Meehl. Construct validity in psychological tests. Psychological Bulletin 52(4): 281–302, 1955. doi:10.1037/h0040957
- de Clippel et al. 2008Geoffroy de Clippel, Hervé Moulin, Nicolaus Tideman. Impartial division of a dollar. Journal of Economic Theory 139(1): 176–191, 2008. doi:10.1016/j.jet.2007.06.005
- Diehl & Wilson 2025Hannah Diehl, Ashia C. Wilson. Semivalue-Based Data Valuation Is Arbitrary and Gameable. arXiv, 2025. arXiv:2506.12619 (preprint)
- DrivenData 2026DrivenData. Sustainable Industry: Rinse Over Run (Teams section of competition rules). DrivenData, 2026 (accessed 2026-11-13). drivendata.org/competitions/56/…/page/137
- Erdős WikiTerence Tao, et al. AI contributions to Erdős problems. teorth/erdosproblems GitHub Wiki (data through 2026-06-30, frozen thereafter), 2025–2026. github.com/teorth/erdosproblems/wiki/…
- Eswaran & Kotwal 1984Mukesh Eswaran, Ashok Kotwal. The Moral Hazard of Budget-Breaking. RAND Journal of Economics 15(4): 578–581, 1984. ideas.repec.org/a/rje/randje/…
- Faigle & Kern 1992Ulrich Faigle, Walter Kern. The Shapley Value for Cooperative Games under Precedence Constraints. International Journal of Game Theory 21(3): 249–266, 1992. doi:10.1007/BF01258278
- Foerster et al. 2018Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, et al. Counterfactual Multi-Agent Policy Gradients. Proceedings of the AAAI Conference on Artificial Intelligence 32(1): 2974–2982, 2018. doi:10.1609/aaai.v32i1.11794
- Ghorbani & Zou 2019Amirata Ghorbani, James Zou. Data Shapley: Equitable Valuation of Data for Machine Learning. ICML, PMLR 97: 2242–2251, 2019. arXiv:1904.02868
- Git AI 2026Git AI. How Git AI works. Git AI Docs, 2026 (accessed 2026-11-13). usegitai.com/docs/cli/how-git-ai-works
- GitHub Docs 2026GitHub. Viewing a project's contributors. GitHub Docs, 2026 (accessed 2026-11-13). docs.github.com/…/viewing-a-projects-contributors
- Goel et al. 2025S. Goel, et al. Great Models Think Alike and this Undermines AI Oversight. ICML, PMLR 267: 19621–19678, 2025. arXiv:2502.04313
- Goodhart 1975C. A. E. Goodhart. Problems of Monetary Management: The U.K. Experience. Reserve Bank of Australia, Papers in Monetary Economics, Vol. I (Sydney conference paper, July 1975, collected 1976; reprinted in Goodhart 1984, Monetary Theory and Practice, Palgrave), 1975. doi:10.1007/978-1-349-17295-5_4 (partially unverified)
- Gowers & Nielsen 2009Timothy Gowers, Michael Nielsen. Massively collaborative mathematics. Nature 461(7266): 879–881, 2009. doi:10.1038/461879a
- HackerOne 2026HackerOne. Collaboration. HackerOne Help Center, 2026 (accessed 2026-11-13). docs.hackerone.com/en/articles/8457618-collaboration (partially unverified)
- Hardt et al. 2016Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, et al. Strategic Classification. ITCS, 2016. arXiv:1506.06980
- Holmström & Milgrom 1991Bengt Holmström, Paul Milgrom. Multitask Principal-Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design. Journal of Law, Economics, & Organization 7 (Special Issue): 24–52, 1991. jstor.org/stable/764957
- Holmström 1982Bengt Holmström. Moral Hazard in Teams. Bell Journal of Economics 13(2): 324–340, 1982. ideas.repec.org/a/rje/bellje/…
- ICMJE 2026International Committee of Medical Journal Editors (ICMJE). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. ICMJE (updated January 2026), 2026. icmje.org/recommendations
- Janzing et al. 2020Dominik Janzing, Lenon Minorics, Patrick Blöbaum. Feature Relevance Quantification in Explainable AI: A Causal Problem. AISTATS, PMLR 108: 2907–2916, 2020. arXiv:1910.13413
- Jiang et al. 2025Z. Jiang, J. Zhang, N. Weir, et al. Core: Robust Factual Precision with Informative Sub-Claim Identification. Findings of ACL: 19833–19856, 2025. doi:10.18653/v1/2025.findings-acl.1018
- Jiang et al. 2026Pengyi Jiang, Xiaoguang Zhu, Quanyan Zhu. Semantic Cooperative Games for Contribution Attribution in LLM-Based Multi-Agent Systems. arXiv, 2026. arXiv:2607.18255 (preprint)
- Kumar et al. 2020I. Elizabeth Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, et al. Problems with Shapley-Value-Based Explanations as Feature Importance Measures. ICML, PMLR 119, 2020. arXiv:2002.11097
- Lee et al. 2025aKiljae Lee, Ziqi Liu, Weijing Tang, et al. Faithful Group Shapley Value. NeurIPS, 2025. arXiv:2505.19013
- Lee et al. 2025bY. Lee, J. Kim, J. Kim, et al. CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists. EMNLP: 15771–15798, 2025. doi:10.18653/v1/2025.emnlp-main.796
- Lee et al. 2026Kiljae Lee, Ziqi Liu, Weijing Tang, et al. Priority-Aware Shapley Value. arXiv, 2026. arXiv:2602.09326 (preprint)
- Li et al. 2021Jiahui Li, Kun Kuang, Baoxiang Wang, et al. Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning. KDD, 2021. doi:10.1145/3447548.3467420
- Li et al. 2026Yanming Li, Xuelin Zhang, WenJie Lu, et al. Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System. arXiv, 2026. arXiv:2602.08335 (preprint)
- Lin 2026Z. Lin. Hidden Prompts in Manuscripts Exploit AI-Assisted Peer Review. Communications of the ACM 69(7): 53–56, 2026. doi:10.1145/3779116
- Linux 2026Linux Kernel Community. AI Coding Assistants. docs.kernel.org (read at 7.4.0-rc2; the Assisted-by tag required in 7.1–7.3 has been removed), 2026 (accessed 2026-11-13). docs.kernel.org/process/coding-assistants.html
- Liu et al. 2024Zijun Liu, Yanzhe Zhang, Peng Li, et al. A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration. COLM, 2024. arXiv:2310.02170
- Li, Ren & Yan 2026Ed Li, Junyu Ren, Cat Yan. Scaling Multiagent Systems with Process Rewards. arXiv, 2026. arXiv:2601.23228 (preprint)
- Lu et al. 2026Mingyu Lu, Yushan Huang, Chris Lin, et al. Agents that Matter: How Subtle Choices Shape Agent Attribution in Multi-Agent Systems. arXiv, 2026. arXiv:2605.27621 (preprint)
- Ma et al. 2026Ming Ma, Jue Zhang, Fangkai Yang, et al. DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems. ICLR, 2026 (arXiv 2025). arXiv:2512.06749
- Merton 1968Robert K. Merton. The Matthew Effect in Science. Science 159(3810): 56–63, 1968. doi:10.1126/science.159.3810.56
- Messick 1995S. Messick. Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist 50(9): 741–749, 1995. doi:10.1037/0003-066X.50.9.741 (partially unverified)
- Messing 2026S. Messing. Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking. arXiv (author note: accepted to NeurIPS 2026), 2026. arXiv:2604.11581 (preprint)
- METR 2025Sydney Von Arx, Lawrence Chan, Beth Barnes. Recent Frontier Models Are Reward Hacking. METR Blog, 2025. metr.org/blog/2025-06-05-recent-reward-hacking
- Mohapatra & Pruthi 2026Arati Mohapatra, Danish Pruthi. Expectations and Practices around AI Disclosure in CS Research. arXiv (accepted to Findings of EMNLP 2026), 2026. arXiv:2608.23271 (preprint)
- Musco & Witter 2025Christopher Musco, R. Teal Witter. Provably Accurate Shapley Value Estimation via Leverage Score Sampling. ICLR, 2025. arXiv:2410.01917
- Nature 2023Nature (Editorial). Tools such as ChatGPT threaten transparent science; here are our ground rules for their use. Nature 613: 612, 2023. doi:10.1038/d41586-023-00191-1
- Nielsen 2008Michael Nielsen. The Future of Science. APS News, November 2008, The Back Page; blog version of the same title. APS News 2008-11 (PDF); Archived blog post
- NISO 2022NISO. CRediT, Contributor Roles Taxonomy (ANSI/NISO Z39.104-2022). NISO Standard, 2022. credit.niso.org
- Optimism 2026Optimism. Season 9: From Experiment to Organization. Optimism Blog, 2026. optimism.io/blog/season-9-from-experiment-to-organization
- Perdomo et al. 2020Juan C. Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, et al. Performative Prediction. ICML, 2020. arXiv:2002.06673
- Polymath 2009D. H. J. Polymath. A new proof of the density Hales-Jewett theorem. Annals of Mathematics 175(3): 1283–1327, 2012 (arXiv:0910.3926 posted 2009). doi:10.4007/annals.2012.175.3.6
- Protocol Guild 2026Protocol Guild. Protocol Guild Annual Report 2025. Protocol Guild Blog, 2026. protocolguild.org/blog/20260129-annual-report-2025
- Rashid et al. 2018Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, et al. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. ICML, PMLR 80: 4295–4304, 2018. arXiv:1803.11485
- Shapin 1989Steven Shapin. The Invisible Technician. American Scientist 77(6): 554–563, 1989. Harvard DASH scan hdl:1/3425945
- Shapley 1953Lloyd S. Shapley. A Value for n-Person Games. In H. W. Kuhn, A. W. Tucker (eds.), Contributions to the Theory of Games II, Annals of Mathematics Studies 28: 307–317, Princeton University Press, 1953. doi:10.1515/9781400881970-018
- Silberzahn et al. 2018R. Silberzahn, et al. Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. Advances in Methods and Practices in Psychological Science 1(3): 337–356, 2018. doi:10.1177/2515245917747646 (partially unverified)
- Simonsohn et al. 2020U. Simonsohn, J. P. Simmons, L. D. Nelson. Specification curve analysis. Nature Human Behaviour 4(11): 1208–1214, 2020. doi:10.1038/s41562-020-0912-z (partially unverified)
- SourceCred 2022ezraU. SourceCred (The Organization) Is Winding Down. SourceCred Discourse Forum, 2022. discourse.sourcecred.io/t/…/1383
- Stenberg 2026Daniel Stenberg. The end of the curl bug-bounty. daniel.haxx.se (personal blog), 2026. daniel.haxx.se/blog/…/the-end-of-the-curl-bug-bounty
- Stevens 1946S. S. Stevens. On the Theory of Scales of Measurement. Science 103(2684): 677–680, 1946. doi:10.1126/science.103.2684.677
- Strathern 1997Marilyn Strathern. 'Improving ratings': audit in the British University system. European Review 5(3): 305–321, 1997.
- Sturmfels et al. 2020Pascal Sturmfels, Scott Lundberg, Su-In Lee. Visualizing the Impact of Feature Attribution Baselines. Distill, 2020. doi:10.23915/distill.00022
- Sundararajan & Najmi 2020Mukund Sundararajan, Amir Najmi. The Many Shapley Values for Model Explanation. ICML, PMLR 119: 9269–9278, 2020. arXiv:1908.08474
- Sunehag et al. 2018Peter Sunehag, Guy Lever, Audrunas Gruslys, et al. Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward. AAMAS Extended Abstract (3 pp., from p. 2085), 2018. arXiv:1706.05296
- SWE-bench 2025Jacob Kahn. Repo State Loopholes During Agentic Evaluation. SWE-bench GitHub issue #465, 2025. github.com/SWE-bench/SWE-bench/issues/465
- Wang & Jia 2023Jiachen T. Wang, Ruoxi Jia. Data Banzhaf: A Robust Data Valuation Framework for Machine Learning. AISTATS, PMLR 206: 6388–6421, 2023. arXiv:2205.15466
- Wang et al. 2022Jianhong Wang, Yuan Zhang, Yunjie Gu, et al. SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning. Advances in Neural Information Processing Systems 35, 2022. arXiv:2105.15013
- Zhang 2026Chenchen Zhang. From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models. arXiv, 2026. arXiv:2604.09459 (preprint)
- Zhang et al. 2025Shaokun Zhang, Ming Yin, Jieyu Zhang, et al. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. ICML, PMLR 267: 76583–76599, 2025. arXiv:2505.00212
- Zhuge et al. 2025M. Zhuge, C. Zhao, D. Ashley, et al. Agent-as-a-Judge: Evaluate Agents with Agents. ICML, PMLR 267: 80569–80611, 2025. arXiv:2410.10934
