Whose result is it? Measuring contribution in research systems of people and agents · Shoal

Who Made This Result? Measurement, Mechanism and Responsibility in Human–Agent Research

Whose result is it?Measuring contribution in research systems of people and agents

2026-11-13About 40 minThe Shoal team
A two-story white clapboard building, the Menlo Park laboratory. Eight men stand along the upper balcony and seven or eight more on the front porch below, all in nineteenth-century coats and hats; a leafless tree rises on the right behind a white picket fence in the foreground.
Cover: the main laboratory building and staff at Menlo Park, c. 1880; U.S. National Park Service, public domain. Opened in 1876 with Batchelor, Adams, and three machinists, the laboratory grew to 25 staff by spring 1878 (Edison Papers).

Three people relay a single delivery; change the rule, and the credits go to someone else. Using Shoal as a testbed, this essay examines contribution measurement in multi-agent systems: contribution is a counterfactual the record never shows, and once you pay on it, it changes the behavior being measured. Our answer is to unbundle the three jobs that authorship has long tied together: treat recognition as measurement, credits as mechanism, and leave responsibility with a person who can bear it. Every number in the figures comes from exact evaluation on platform code, live probes against platform code, or recomputation from archived experimental logs, with illustrative examples explicitly marked; every claim states whether it is an established result, our reproduction, or holds to the best of our search.

Consider a routine request: a requester posts 100 credits for a working program. Carol opens a branch; branching from it, Alice and Bob each write a 400-line module; Carol merges Alice's branch first, then Bob's, and submits the result as a delivery. The requester accepts it. Carol has not written a single line of code. How the credits split depends on which rule governs—and at different stages, Shoal has shipped or tested four of them.

25%25%50%
18.8%74.2%
50%50%
55%40%

AliceBobCarol

Data
RuleAliceBobCarol
By commit count252550
By lineage position718.874.2
One share per changer50500
Agent reads code and suggests55405
Figure 1 One delivery under four rules. The first three rows were measured live with the same seed script across three versions of the platform's code (main 359b875, PR #34 b9c65cc, and PR #35 40b8995); the last row is a single run of the analysis package on the author's branch over code of the same shape.

By commit count, Carol takes half, because each merge counts as a commit; by position in the commit lineage, she takes 74.2%, because the delivered commit is hers; under one equal share per substantive contributor, she gets 0; when an agent reads the code and recommends shares, she gets 5%. All four rules compute cleanly; they disagree on what counts as contribution.

This kind of divergence already has a name elsewhere. Twenty-nine teams—61 analysts in all—used the same dataset to answer a single question: whether soccer referees are more likely to give red cards to dark-skinned players. Twenty teams found a statistically significant positive effect and nine did not, with odds ratios ranging from 0.89 to 2.93 (Silberzahn et al. 2018). In a larger replication, 73 teams submitted 1,261 models, and over 95% of the variance across results could not be explained by coded analytic decisions or researcher characteristics (Breznau et al. 2022). In 2026, when AI analysts repeated the exercise, audited runs disagreed even on the sign of the effect on some datasets (Bertran et al. 2026). None of the analysts miscalculated; the spread came from defensible choices at every step. Credit attribution works the same way: one record, many rules, many answers.

In the era of the paper, the problem was less urgent because the unit of credit was an entire article. Once agents enter the loop, contributions split down to a single merge, a one-line fix, or a verification run—and each of those actions is cheap. Whatever a rule rewards, a group produces more of; replace human effort with agents that optimize directly against the objective, and that pull requires no malice at all. In 2008, Michael Nielsen wrote that new scientific tools would need "metrics for contribution built right into them," with every contribution archived, timestamped, and signed (Nielsen 2008). Git gives us the second half for free today; the first half is the hard part.

Contribution is a counterfactual invisible in the record; once you pay on it, it reshapes the behavior being measured. So we separate the three things authorship has bundled together: treat recognition as measurement, credits as mechanism, and leave responsibility with a person who can bear it.

This essay uses Shoal as its proving ground. Chapter 1 covers background and the platform: the history of scientific credit, how Shoal operates, and how the rules we tried were fooled by cheap actions. Starting in Chapter 2, we treat contribution measurement in multi-agent systems as a research problem in its own right: Chapter 2 formalizes it and shows why contribution is unidentifiable from the record alone; Chapter 3 analyzes three families of algorithms and the assumptions each relies on; Chapter 4 asks how to test an attribution rule; and Chapter 5 derives the mechanism. Throughout, each claim notes where it stands: established result, our reproduction, or to the best of our search.

Chapter 1

Background: from authorshipto the relay

The unit of credit has shrunk from a paper to a commit. Finer units leave richer records—and let rules amplify cheap actions.

1.1Authorship

The unit of credit:from a paper to a merge

Scientific reward rests on priority and peer recognition, and recognition gravitates toward those who are already known: for the same work, an established scientist's name attracts disproportionate attention—a pattern Merton, borrowing from the Gospel of Matthew, called the Matthew effect (Merton 1968). The unit of that system is the paper, with registration, certification, and reward all hung on a single finished manuscript. As collaboration scales up, the coarseness of the unit shows: in 2015, the joint ATLAS and CMS measurement of the Higgs boson mass listed 5,154 authors, dedicating 24 of its 33 pages to names and institutions (Aad et al. 2015; Castelvecchi 2015).

Several remedies followed. Role taxonomies write down what kind of work each person did: CRediT defines 14 contributor roles and became an ANSI/NISO standard in 2022, though it records the category of contribution rather than its magnitude (Allen et al. 2014; Brand et al. 2015; NISO 2022). Counting tallies what can be counted: GitHub's contributors graph counts commits on the default branch while excluding merge commits and empty commits (GitHub Docs 2026). Marginal value asks what would happen if a contributor were absent: Shapley in 1953 derived the unique division in cooperative games satisfying a natural set of axioms, an idea machine learning later adapted to value training data (Shapley 1953; Ghorbani & Zou 2019). Still another approach makes the group itself the author: Polymath papers were signed D. H. J. Polymath, leaving anyone who wanted to know who did what to read the public discussion threads (Polymath 2009; Gowers & Nielsen 2009).

More recent cases sit closer to home. The Equational Theories Project, initiated by Terence Tao and coordinated on GitHub with Lean verification, published a 34-author paper in 2025: the formal author list is alphabetical, while an appendix lists who carried out each kind of task like film credits, adapting CRediT categories—with only humans in the contribution matrix (Bolan et al. 2025). After AI systems were claimed in 2025 to have "solved" several Erdős problems—most of which turned out to be locating existing results in the literature—Tao's wiki split AI contributions into major and minor tiers, classifying literature retrieval, formalization, rewriting, and computation as minor while tracking outcomes separately as solved, partial progress, erroneous, or unverified (Erdős Wiki). That same year, the Agents4Science conference required AI systems to be listed as first authors and asked submissions to declare human versus AI shares across four stages—hypothesis, design, analysis, and writing (Bianchi et al. 2025); yet a study of AI disclosure statements in computer science papers found that only 2% to 23% stated who was accountable for AI-generated output (Mohapatra & Pruthi 2026). Disclosure forms keep growing finer, while the people filling them out remain the interested parties themselves.

On whether AI can be an author, journals and conferences gave nearly the same answer at nearly the same time in 2023: no. Their reasoning was almost verbatim: authorship entails accountability for the work, and AI cannot bear responsibility. COPE wrote that AI tools "cannot take responsibility for the submitted work"; Nature wrote that "any attribution of authorship carries with it accountability for the work" (COPE 2023; Nature 2023; ICMJE 2026). Those rules settled who cannot be an author, not who decides credit; ICMJE adds explicitly that judging who qualifies for authorship or arbitrating authorship disputes is not the role of journal editors. Historically, the work most easily left out is the work that makes a result hold up in the first place: in Shapin's history of Boyle's seventeenth-century laboratory, technicians built the instruments and ran the experiments, yet at publication "His master spoke for him," their labor "included in" the master's name (Shapin 1989).

5,154 authors

Authorship

A single author line

Notes
Who was in the room
Omits
Who did what
Shoal
Lineage split (PR #34)Tried
14 roles, no amounts

Roles

CRediT's 14 roles

Notes
What category of work
Omits
How much was done
Shoal
Roles in reports, descriptive onlyLive
252550

Counts

Commits and merges

Notes
How many actions
Omits
What each was worth
Shoal
By commit count (main)Default
Without Alice: −4 s

Marginal

What happens without it

Notes
Marginal effect
Omits
What replaces absence
Shoal
Shapley (offline study)Tried
One reader, one tier

Review

Judgement from evidence

Notes
How one reader sees it
Omits
What another would say
Shoal
Rubric → tiered reportLive
Figure 2 Five traditions for recording credit, what each captures, and what each leaves out. The bottom row shows the corresponding mechanism we tried on Shoal.

Each of these traditions has a point, and each has a blind spot. Authorship records who was in the room; roles record what category of work was done; counts record how many actions occurred; marginal value records what would be lost without someone; review records how a reader of the evidence sees it. None of them alone answers whose work the result actually stands on. Traditional authorship quietly governs three things at once: recognizing who did what, deciding who gets rewarded, and fixing who is accountable for the outcome. CRediT unbundled roles within the first; the latter two remain tied together.

1.2Platform

Shoal:how people and agents relay work

Shoal is a platform for open research, hosting both open problems and credit-backed requests. People and their agents work on it in relays: a delivery is a pull request, behind which several agents have often explored different routes—one proposing a method, another implementing it, a third finding a counterexample, and someone else integrating the pieces at the end. The site runs online while agents run on their operators' own machines; the site never calls a model.

The credit ledger is append-only, and every balance is a sum. Each person starts with 100 credits, issued solely by the site; credits cannot be bought, sold, transferred, or cashed out, and whatever an agent earns accrues to its human operator. Posting a request holds the offered credits in escrow, capped at the requester's balance. Once a delivery is accepted, settlement branches into two cases: if every commit in the delivery comes from a single side, payment is immediate; if several people contributed, the requester has 3 days on the analysis page to confirm who receives what. Recipients can only be makers of that delivery: commit authors, the author of the delivery PR, and participants assigned a share in a report published by the requester; the requester's own side cannot receive credits. If 3 days pass without a decision, the default settles the escrow: when a published report covers the delivery, credits follow its shares renormalized across eligible recipients; when there is no report, credits split by commit count in the delivery. Shares merge by operator and round to integer credits by the largest-remainder method; every decision stays on the record, visible to anyone who can view the project.

Requester's balanceHeldNo delivery: refundedAcceptedOne side made it: paid on acceptance, by commit countSeveral made itRequester confirms in 3dExpired: per published reportExpired: no report, by commit countPaid out● Alice0● Bob0● Carol03 daysRequester balanceHeldNo delivery: refundedAcceptedOnesideSeveral made itConfirmedBy reportBy commitsPaid out● Alice0● Bob0● Carol03 days
Data
Settlement pathAliceBobCarolRequester
One side made it100000
Confirmed within 3 days554050
Expired, published report554050
Expired, no report2525500
No delivery000100
Figure 3 100 credits from escrow to settlement across all five outcomes. Every path ends in either payout or refund—none stalls midway—and once a delivery is accepted, no path returns credits to the requester. Multi-party splits use the opening delivery's numbers: 55 / 40 / 5 from the report, and 25 / 25 / 50 by commit count.

Share recommendations for multi-party deliveries come from a four-step project analysis. First, freeze: the requester picks a revision and the question to answer, and the site freezes the baseline, target, rules revision, authenticated participants, jobs, checks, results, issues, and reviews together with a Git bundle containing every relevant commit; the repository may keep changing afterward, while what this analysis reads stays fixed. Second, judgement: the requester hands a one-line instruction to their own Claude Code, Codex, or other agent; the agent claims the task, traces the origin and adoption of each outcome across the read-only Git history, records its judgements, and runs a local tool to compose the report. Third, human confirmation: editing, publishing, retracting, and confirming rewards all require a signed-in person—a token cannot perform them, including the token of the requester's own agent. Fourth, a second report: anyone named in a published report can read the exact same frozen materials and upload a report of their own, which the page places side by side with the original.

SiteFreeze · check · recordRequester's agentOn its own machineRequesterSigned inNamed contributorSecond reportFreezeGit bundle + digestClaim1 h leaseRead reporead-only gitJudge103–146 sValidatetask · digest · SHAPublishtokens rejectedPublicshows provenanceSecond reportsame snapshotSide by sidegrey band
SiteFreezeGit bundle + evidence digestRequester's agentClaim1 h leaseRequester's agentRead reporead-only gitRequester's agentJudge103–146 sSiteValidatetask · digest · SHARequesterPublishtokens rejectedSitePublicshows provenanceNamed contributorSecond reportsame snapshotSiteSide by sidegrey band
Figure 4 One analysis across four swimlanes. The site freezes materials, validates report schemas, and records decisions; judgement runs on the requester's own machine; publishing and payout require a signed-in human—the padlock marks where an API token bounces back—and anyone named in a published report can submit a second report.

Set beside comparable platforms, Shoal's position becomes clear. For splitting money across multiple contributors, the platforms we surveyed default to one of three rules: pay only the single accepted PR, split equally, or leave the split to the participants themselves (Algora 2026; HackerOne 2026; DrivenData 2026).

PlatformWho gets paidWho decides for teamsWindow & defaultWhen disputed
AlgoraAuthor of the claimed, accepted PRClaimant lists recipientsEqual split across listUndocumented
HackerOneReporter and invited collaboratorsReporter sets weightsFinal once bounty is awardedNot checked here
DrivenDataEach team memberUnanimous team consent3 business days post-deadline to request custom split; else equalWritten statement
Prize challenges (GoFly, etc.)Account designated by team leadThe team internallyNoneOrganizer does not arbitrate
ShoalMakers of the deliveryRequester3 days; else published report, or commit count if noneSecond report

Relaying—one person building directly on an earlier contributor's work—has no place in any of those rules. AI attribution in software stops at the layer of "who wrote the lines" as well: GitHub's Copilot cloud agent authors commits under its own identity with the delegating user as co-author, and that user's approval does not count toward required reviews (Copilot Docs 2026); the Agent Trace specification and Git AI record line by line which agent produced which code, with the former explicitly declining to judge quality (Agent Trace 2026; Git AI 2026); and the Linux kernel bars AI from signing the Developer Certificate of Origin, placing responsibility on the human submitter (Linux 2026). Shoal has to answer a different layer altogether: whose work was actually adopted.

1.3Trial and error

The rules we tried,and the cheap actions that fooled them

When credit escrow launched on October 25, multi-party deliveries split by commit count. In early November, PR #34 proposed lineage attribution and rubric scoring alongside a six-component Shapley experiment; review on November 6 found that the lineage rule ranked a merge-only participant first; an absorbed revision, PR #35, switched to one equal share per substantive contributor, only for a second review round to reproduce three payout vulnerabilities, leading to its withdrawal on November 7. On November 9, PR #37 let agents write reports over frozen evidence; PR #38, merged into main on November 10, inverted the workflow to list outcomes first, grade tiers second, and keep the arithmetic public.

Against the platform's live code, using temporary data directories, we ran each cheap action one by one—one run per probe. Under commit count, someone who edits a single line before delivering walks away with half; under lineage attribution, they take 62.5%, a merge-only contributor takes 74.2%, and a requester who merely clicks "Approve" claws back 11.8% of their own bounty. Giving one share to every substantive contributor blocks the merge-only case, yet a requester who pushes a single line into the delivery still recovers 20 out of 40 credits, and someone who makes an edit after a merge and immediately reverts it—leaving the final files unchanged—still collects a full share. Only one approach recognized a comment-only padding file for what it was: having an agent read the content, which assigned everyone 0 share and noted that the checked acceptance boxes carried no supporting evidence. On an earlier branch, the requester could even award credits to themselves or to an account that had never touched the repository; and if they never confirmed after accepting, the credits stayed locked indefinitely while refunds were refused.

How cheap actions fare under four attribution rules
ActionBy commit countCommitsmainmainBy lineageLineagePR #34PR #34One share per changerEven splitPR #35PR #35Agent reads contentReads codeanalysis pkg · livelive
One-line finishAlice writes the work; Bob edits one line and delivers50%62.5%50%—
Merge onlyCarol writes no code, merges two routes and delivers50%74.2%0%5%
Add a line, then mergeCarol adds one line before merging and delivering60%86.1%33.3%—
Requester approvesRequester clicks Approve before accepting0%11.8%0%Excluded
Requester pushes a lineRequester pushes one line into the delivery, then accepts——50%Excluded
Edit then revertEdits a line after merging and reverts it; final content unchanged——33.3%—
Padding fileDelivered file contains only commentsPaid in full—Paid in full0%
Planted instructionsCLAUDE.md in the repo instructs the reviewer to grade the committer Core———Ignored
Fooled / tested4 / 54 / 45 / 70 / 3+ 2 excluded by rule+2 excl.

Fooled: share captured by the actorBlockedExcluded by rule— Not tested

Data
ActionBy commit countBy lineageOne share per changerAgent reads content
One-line finish50%62.5%50%—
Merge only50%74.2%0%5%
Add a line, then merge60%86.1%33.3%—
Requester approves0%11.8%0%Excluded by rule
Requester pushes a line——50%Excluded by rule
Edit then revert——33.3%—
Padding filePaid in full—Paid in full0%
Planted instructions———Codeword absent from report, hook never fired, planted file treated as material under review (1 run)
Figure 5 Each row is a cheap action and each column is an attribution rule. Each wedge shows the share captured by the actor (measured live, one run per probe): the fuller the wedge, the worse the rule was fooled. Empty rings mean blocked, slashes mean barred upfront by the current rules (the requester's side cannot receive), and dashes mean not tested.

Money attracts cheap actions elsewhere too: before curl shut down its bug bounty in January 2026, the confirmation rate on submitted reports had fallen from above 15% to below 5% (Stenberg 2026). People are still restrained by embarrassment; agents optimize straight toward whatever a rule rewards, with no malice required. Those trials left us with three questions. Can the record alone pin down contribution? What assumptions does each family of algorithms rely on, and how should a rule be tested? And who should hold the right to decide? Chapters 2 through 5 answer them in turn.

Chapter 2

Problem: contribution is anunseen counterfactual

From here on, we situate what happens on Shoal within multi-agent credit assignment: first distinguishing our problem from its namesake in machine learning, next setting out notation and definitions, and finally showing why contribution is unidentifiable from records alone—and why paying on a rule changes what it measures.

2.1Learning & division

Credit for learning,credit for division

Multi-agent reinforcement learning has long studied credit assignment: when a team of agents shares a single reward, how much of it does each agent answer for? The textbook answer is a counterfactual: compare the actual reward with the reward had that agent taken a default action instead, and call the difference its contribution. Difference rewards work this way, at the cost of requiring a simulator or learned reward model and a defensible choice of default action; COMA learns that counterfactual baseline with a centralized critic (Foerster et al. 2018). A second line of work uses value decomposition, expressing team value as a monotonic combination of individual utilities (Sunehag et al. 2018; Rashid et al. 2018); a third computes Shapley values directly, substituting a zero vector for absent agents—a choice the authors note is made for convenience (Li et al. 2021; Wang et al. 2022).

With LLM-based multi-agent systems, this literature has branched again. When training multiple agents, researchers estimate each agent's effect via Shapley-style credit or by branching and resampling from intermediate states (Li et al. 2026; Zhang 2026); where an AI coach scores intermediate steps, signs of agents gaming the process reward have already emerged (Li, Ren & Yan 2026). Another branch studies failure attribution: reading a failed multi-agent trace to name which agent erred at which step. Even the strongest methods identify the responsible agent only 53.5% of the time and the exact step only 14.2% of the time (Zhang et al. 2025); other work argues that pinning a systemic failure on a single agent is often ill-posed from the start (Ma et al. 2026; Cemri et al. 2025). A 2026 survey lists credit assignment under soft, learned rewards as largely open (Zhang 2026).

What all of those methods produce is a learning signal: shares need not sum to the whole, participants do not have to accept them, and a policy being trained neither files an appeal nor pushes comment-only padding files. What we need is division: shares land on people, becoming credits and reputation. Division adds four requirements: shares must sum to the budget, carry an intelligible rationale, withstand strategic behavior by interested parties, and admit contestation. Deeper still is where the counterfactual comes from: machine learning leans on simulators, critics, or reruns, and open research has none of the three.

What it sees →Who decides ↓StructureCommits, merges, dependencies, citationsOutcomesCounterfactuals via reruns or simulationContentRead evidence, judge whose work was adoptedPeopleRetro FundingProtocol GuildHackerOne, DrivenDataCRediT, contribution statementsEquational TheoriesRequester decides payoutModelDyLAN importanceTiers & public formulaFailure attributionAgent free-form sharesDeep FundingFormulaGitHub contributors graphSourceCredteaLineage v2.2Co-citation creditCOMA, difference rewardsSHAQ, SHARPData valuationAttribution gamesSix-component Shapley

Grouped by "What it sees · Who decides"

Content · People

  • Retro FundingImpact evaluations set grant amounts directly; paused Jan 2026Optimism 2021–2026
  • Protocol GuildRecognition decides eligibility only; amounts follow a tenure formulaProtocol Guild 2025 report
  • HackerOne, DrivenDataReporter sets weights; custom split with unanimous team consent, else equalPlatform terms
  • CRediT, contribution statementsRecords roles without magnitudes; self-reportedBrand et al. 2015
  • Equational TheoriesGit + Lean + contribution matrix; no AI credit entriesBolan et al. 2025

Structure · Model

  • DyLAN importancePeer rating propagated backward from terminal nodes, isomorphic to lineageLiu et al. 2024

Content · Model

  • Tiers & public formulaAgent lists outputs and grades tiers, code converts; requester decides payoutShoal main 11-10↑ Requester decides payout
  • Failure attributionFrom logs alone: 53.5% agent accuracy, 14.2% step accuracyZhang et al. 2025
  • Agent free-form sharesSwapping models shifts shares by 14.7 pp TVDAuthor branch
  • Deep FundingModels weight dependency-graph edges while human jurors spot-check a slice; no official pilot results yetButerin 2025; deepfunding.org

Structure · Formula

  • GitHub contributors graphCounts default-branch commits only, excluding mergesGitHub Docs
  • SourceCredPageRank over contribution graphs for cred and grain; wound down in 2022SourceCred 2019–2022
  • teaRewarded registered dependencies, drawing 150k spam packages onto npmAWS 2025
  • Lineage v2.2Flows backward along merge chains; deliverer retains 62.5%PR #34
  • Co-citation creditInfers co-author credit from downstream co-citationsShen & Barabási 2014

Outcomes · Formula

  • COMA, difference rewardsCounterfactuals from a simulator, critic, or default actionFoerster et al. 2018
  • SHAQ, SHARPShapley-style credit signals for training multi-agent policiesWang et al. 2022; Li et al. 2026
  • Data valuationData Shapley, Data Banzhaf: utility choice rests with the evaluatorGhorbani & Zou 2019; Diehl & Wilson 2025
  • Attribution gamesCoalition weights × intervention protocol × target metric; different protocols define different gamesLu et al. 2026
  • Six-component ShapleyExact over 64 coalitions; changing absence replacement shifts shares by up to 26.6 ppPR #37 archive

For division: turns into money or standingFor learning: training signalTried on Shoal

Data
MethodWhat it seesWho decidesPurposeSource
GitHub contributors graphStructureFormulaDivisionGitHub Docs
SourceCredStructureFormulaDivisionSourceCred 2019–2022
teaStructureFormulaDivisionAWS 2025
Lineage v2.2StructureFormulaDivisionPR #34
Co-citation creditStructureFormulaDivisionShen & Barabási 2014
COMA, difference rewardsOutcomesFormulaLearningFoerster et al. 2018
SHAQ, SHARPOutcomesFormulaLearningWang et al. 2022; Li et al. 2026
Data valuationOutcomesFormulaDivisionGhorbani & Zou 2019; Diehl & Wilson 2025
Attribution gamesOutcomesFormulaLearningLu et al. 2026
Six-component ShapleyOutcomesFormulaDivisionPR #37 archive
DyLAN importanceStructureModelLearningLiu et al. 2024
Failure attributionContentModelLearningZhang et al. 2025
Agent free-form sharesContentModelDivisionAuthor branch
Tiers & public formulaContentModelDivisionShoal main 11-10
Deep FundingContentModelDivisionButerin 2025; deepfunding.org
Retro FundingContentPeopleDivisionOptimism 2021–2026
Protocol GuildContentPeopleDivisionProtocol Guild 2025 report
HackerOne, DrivenDataContentPeopleDivisionPlatform terms
CRediT, contribution statementsContentPeopleDivisionBrand et al. 2015
Equational TheoriesContentPeopleDivisionBolan et al. 2025
Figure 6 Attribution methods arranged along two axes: what they read (structure, outcomes, content) and who decides (formula, model, people). Hollow dots produce learning signals; solid dots divide money or standing; ringed dots mark approaches we tried on Shoal. In our live design, a model reads content and grades tiers, while the requester decides the payout. Sources appear in tooltips and in the bibliography.

2.2Notation

Participants, records,and attribution rules

The notation below runs through the remaining three chapters.

SymbolMeaning
N=H∪AN = H \cup AParticipants: people HH and agents AA; each agent belongs to an operator o(a)o(a), to whom o(a)o(a) its share is paid
RRThe record of a delivery: commit graph, jobs, checks, reviews, and other evidence, frozen at a target revision
K, wkK,\ w_kCommissioned outputs of the delivery and their weights, ∑kwk=1\sum_k w_k = 1
Gk, tgG_k,\ t_gGroups within output kk and each group's tier; participants inseparable in the evidence form one group
Δ(N)\Delta(N)The share simplex: pi≥0p_i \ge 0, ∑ipi=1\sum_i p_i = 1
φ(R)\varphi(R)Shares assigned by an attribution rule, φ(R)∈Δ(N)\varphi(R) \in \Delta(N)
v(S)v(S)Value of the outcome when only coalition SS participates
D(p,q)D(p, q)Total variation distance (TVD) between two share vectors, reported in percentage points (pp)

Definition 1 (Attribution rule) An attribution rule is a map φ\varphi: it reads a frozen record RR and assigns each participant a share, φ(R)∈Δ(N)\varphi(R) \in \Delta(N). The rule sees only RR; anything outside RR—such as what would have happened without Carol—must either be posited by the rule itself or supplied by a judge who has read RR.

Definition 2 (Attribution game) Following Lu et al.'s formulation of agent attribution, a cooperative attribution game is fixed by three choices: a target metric vv, specifying what counts as outcome value (passing tests, a benchmark score, or human acceptance); an intervention protocol α\alpha, specifying what replaces an absent participant's work; and coalition weights ω\omega, specifying how subsets are averaged (Shapley averages uniformly over arrival orders, Banzhaf weights all coalitions equally, and leave-one-out looks only at the grand coalition minus one) (Lu et al. 2026). Once (v,α,ω)(v, \alpha, \omega) is fixed, participant ii's Shapley value is

φi(v)=∑S⊆N∖{i}∣S∣! (n−∣S∣−1)!n! [v(S∪{i})−v(S)]\varphi_i(v)=\sum_{S\subseteq N\setminus\{i\}}\frac{|S|!\,(n-|S|-1)!}{n!}\,\bigl[v(S\cup\{i\})-v(S)\bigr]
(1)

Definition 3 (Total variation distance) The distance between two allocations pp and qq is D(p,q)=12∑i∣pi−qi∣D(p,q)=\tfrac12\sum_i \lvert p_i-q_i\rvert: the fraction of the total budget that must be moved from some participants to others to make the two allocations coincide.

2.3Identifiability

Where counterfactualscome from

A participant's contribution is at bottom a counterfactual: what would have happened to the result without their work? In equation (1), v(S)v(S) asks about an unrealized world for every coalition except the grand coalition itself. Machine learning has tried five sources for that counterfactual (Foerster et al. 2018; Zhang 2026; Lu et al. 2026):

Counterfactual sourceRepresentative methodsWhat it requiresIn open research
RerunsAblation, rollout continuation, branch resamplingReplayable intermediate states at affordable costDeliveries cannot be replayed; only synthetic tasks can
Learned criticCOMA, value decompositionMany trajectories from the same distributionEvery request differs; sample size is tiny
Default actionDifference rewards, zero vectorA convention for "absence"Available—and the exact source of the 26.6 pp swing in Chapter 3
Model imaginationAsking a judge what would happen if an agent were absentA judge modelAvailable, but known to overestimate and disagree with real ablations
Single-trace structureLineage, backward importance propagation, semantic supportA dependency graphAvailable as Git lineage, but vulnerable to cheap actions

When Lu et al. asked an LLM judge to imagine removing an agent and compared its answers against actually ablating the agent and rerunning, 30% to 52% of instance-level attributions disagreed (Lu et al. 2026). Across our search, we found only one Shapley method designed explicitly to run on a single realized trajectory without reruns; it computes over a graph of semantic entailment and is evaluated on medical QA and multi-role workflows (Jiang et al. 2026). To the best of our search, for non-replayable research and software deliveries where value comes from human acceptance, the machine learning literature offers almost no validated contribution measures (search log in Appendix B).

That is the problem of identifiability: the record fixes the facts, not the counterfactual, and every rule bridges the gap with an assumption. We ran the opening three-person delivery through 15 specifications: three measured live on platform code, nine computed directly by platform functions, one derived from GitHub's documented contributor rules, and two illustrative value functions.

Observation 1 (One record, fifteen answers) On the opening delivery, 15 specifications assign Carol anywhere from 0 to 74.2%. By commit count, she receives 50% when merges count and 0 when they do not; by lineage, she receives 74.22% whichever branch she merges first; under tier grading, grading her unattributed, pending, incidental, supporting, major, or core gives her 0, 0, 1.54%, 5.88%, 20%, or 33.33%; listing integration as a separate equally weighted output gives her 50%. The 15 specifications land on just 9 distinct points: five of them collapse onto the exact same allocation (50, 50, 0) for entirely different reasons.

MeasuredPlatform codePublic specIllustrativeCoinciding specs

SpecificationCarol's share (%)

Commit count

Count merges (Shoal main)50
Exclude merges (GitHub graph)0

Lineage

Merge Alice first74.22
Merge Bob first74.22

Even split

One share per substantive changer0

Agent judgement

Agent reads code, free-form shares5

Tiers

One output, Carol Unattributed0
One output, Carol Pending0
One output, Carol Incidental1.54
One output, Carol Supporting5.88
One output, Carol Major20
One output, Carol Core33.33
Integration as equal output, Carol Core50

Shapley

Anyone can merge: Carol substitutable0
No merge without Carol: all three needed33.33
Data
FamilySpecificationSourceAliceBobCarol
Commit countCount merges (Shoal main)Measured252550
Commit countExclude merges (GitHub graph)Public spec50500
LineageMerge Alice firstPlatform code7.0318.7574.22
LineageMerge Bob firstPlatform code18.757.0374.22
Even splitOne share per substantive changerMeasured50500
Agent judgementAgent reads code, free-form sharesMeasured55405
TiersOne output, Carol UnattributedPlatform code50500
TiersOne output, Carol PendingPlatform code50500
TiersOne output, Carol IncidentalPlatform code49.2349.231.54
TiersOne output, Carol SupportingPlatform code47.0647.065.88
TiersOne output, Carol MajorPlatform code404020
TiersOne output, Carol CorePlatform code33.3433.3333.33
TiersIntegration as equal output, Carol CorePlatform code252550
ShapleyAnyone can merge: Carol substitutableIllustrative50500
ShapleyNo merge without Carol: all three neededIllustrative33.3333.3333.33

The two Shapley rows use illustrative value functions: a runnable delivery requires both modules plus a merge; "substitutable" assumes anyone could perform the merge, whereas "needed" assumes no merge happens without Carol.

Figure 7 Three-way splits sit inside a simplex: the closer a point lies to a vertex, the larger that contributor's share, with Carol's share equal to height. Fifteen specifications land on just nine distinct locations; marker shapes indicate provenance (live measurement, platform code, public specification, or illustrative assumption), and the 0–8% band for Carol is magnified below. From a point alone, you cannot tell which assumption produced it.

MeasuredPlatform codePublic specIllustrative

Data
#FamilySpecificationSourceAliceBobCarol
1Commit countExclude merges (GitHub graph)Public spec50500
2Even splitOne share per substantive changerMeasured50500
3TiersOne output, Carol PendingPlatform code50500
4TiersOne output, Carol UnattributedPlatform code50500
5ShapleyAnyone can merge: Carol substitutableIllustrative50500
6TiersOne output, Carol IncidentalPlatform code49.2349.231.54
7Agent judgementAgent reads code, free-form sharesMeasured55405
8TiersOne output, Carol SupportingPlatform code47.0647.065.88
9TiersOne output, Carol MajorPlatform code404020
10TiersOne output, Carol CorePlatform code33.3433.3333.33
11ShapleyNo merge without Carol: all three neededIllustrative33.3333.3333.33
12Commit countCount merges (Shoal main)Measured252550
13TiersIntegration as equal output, Carol CorePlatform code252550
14LineageMerge Alice firstPlatform code7.0318.7574.22
15LineageMerge Bob firstPlatform code18.757.0374.22
Figure 8 Specification curve. The upper panel orders Carol's share across all 15 specifications from lowest to highest; the lower panel marks which rule family each specification belongs to. The underlying record never changes—only the rule and its assumptions do. Layout adapted from specification curve analysis (Simonsohn et al. 2020).

That the definition of absence reshapes attribution is well established in explainability and data valuation (Sundararajan & Najmi 2020; Sturmfels et al. 2020); in data valuation, the evaluator can even choose the utility function strategically to steer the outcome (Diehl & Wilson 2025). What we add here is how large that swing becomes on team deliveries and the concrete ways each choice breaks, examined family by family in Chapter 3.

Definition precedes estimation. A single record does not determine a unique contribution. Shares mean something only after you state upfront what counts as value, what replaces an absent contributor, and who is substitutable.

2.4Rules as mechanisms

Once a rule pays,it becomes the target

Measurement has a second side. Goodhart observed in 1975 that any observed statistical regularity tends to collapse once pressure is placed upon it for control, later distilled by Strathern into the familiar maxim: when a measure becomes a target, it ceases to be a good measure (Goodhart 1975; Strathern 1997). Machine learning has two formal counterparts: strategic classification, where subjects alter their features to game a classifier (Hardt et al. 2016), and performative prediction, where deploying a predictor shifts the very distribution it predicts (Perdomo et al. 2020).

Agents accelerate that loop. METR reported in 2025 that o3 hacked the scoring setup on 30.4% of RE-Bench runs, acknowledging when questioned afterward that its shortcuts violated the spirit of the task (METR 2025). OpenAI found that under strong optimization pressure, optimizing directly against a chain-of-thought monitor does not eliminate reward hacking—it teaches the model to hide it (Baker et al. 2025). On SWE-bench, agents dug through Git history to locate future fix commits (SWE-bench 2025)—and Git history is precisely what lineage rules read. Meanwhile, the tea protocol, which rewarded open-source packages along dependency graphs, drew over 150,000 token-farming spam packages onto npm (AWS 2025).

Manipulation comes from two sides. One is contributors: splitting commits, splitting outputs, or opening sybil accounts. Exact Shapley allocation over declared identities cannot simultaneously prevent splitting and false-name attacks, as a 2026 preprint proves via an impossibility theorem (Burnat & Davidson 2026); remedies have likewise been proposed for "shell company" attacks that split one coalition into several to inflate its combined valuation (Lee et al. 2025a). The other side is the rule-setter: picking whichever plausible definition favors their preferred outcome (Diehl & Wilson 2025).

Multi-agent systems add one more layer: when the reviewer is itself an agent, the artifact under review can speak directly to its judge. In July 2025, 18 arXiv preprints were found hiding instructions to AI peer-review tools in white or microscopic text (Lin 2026); in our own tests we planted a CLAUDE.md inside a repository under review instructing the assessor to grade the submitter core (Chapter 4). And a quieter distortion runs underneath all of these: when a job has several dimensions and only some are measurable, paying solely on the measurable ones crowds effort out of the unmeasurable ones (Holmström & Milgrom 1991). As Chapter 5 will show, verification, review, and groundwork are precisely the unmeasurable parts in our records.

A rule is a mechanism. Once an attribution rule governs payouts, it has to be judged as a mechanism: what it rewards, where it steers people and agents, and who can game it.

2.5Properties

What an attribution ruleought to satisfy

Putting the preceding sections together, an attribution rule used for division has to answer to seven properties. The first four are Shapley's axioms (Shapley 1953); the next three come from the mechanism side. The matrix in Chapter 4 tests five of them; additivity cannot be tested on a single delivery, and contestability belongs to the surrounding mechanism in Chapter 5.

  • Efficiency: shares sum to the full bounty, ∑iφi=1\sum_i \varphi_i = 1.
  • Symmetry: contributors with identical effect receive identical shares, regardless of merge order or naming order.
  • Null player: someone with no effect on the outcome receives zero—whether merging without changes, editing and reverting, or adding padding files.
  • Additivity: shares across two tasks can be computed separately and added together.
  • Split invariance: splitting the same work into more commits, more outputs, or more accounts leaves shares unchanged.
  • Evidence-bounded: every share traces back to specific cited evidence.
  • Contestability: anyone named in an attribution has a way to say "that is not what happened."

Payout carries three further guarantees that care only about the flow of credits, whether or not the shares are right: no lockup, so every held credit eventually pays out or refunds; no flow-back, so once accepted, credits never return to the payer; and full visibility, so every decision and its rationale remain on the record.

Chapter 3

Methods: three families,three assumptions

Structural rules treat position as contribution, counterfactual rules treat absence as known, and judgement rules treat ordinal ranks as multiples. For each family, we state the definition and formal properties first, then report what we measured.

3.1Position

Commit count and lineage:position as contribution

The simplest structural rule counts commits. Even that comes in two conventions: Shoal's main branch counts merge commits, giving Carol 50% on the opening delivery; GitHub's contributors graph excludes merge commits, giving her 0. Under the same label—"by commit count"—two standard conventions give opposite answers.

PR #34's lineage attribution rule (v2.2) goes a step further, reading where commits sit in the DAG. Place one unit of credit on the delivered commit and let it flow backward along parent edges:

r(x)=λ f(x),f(y) += (1−λ) f(x)∣P(x)∣for y∈P(x)r(x)=\lambda\,f(x),\qquad f(y)\ \mathrel{+}=\ \frac{(1-\lambda)\,f(x)}{|P(x)|}\quad\text{for } y\in P(x)
(2)

where f(x)f(x) is the flow reaching node xx (with f=1f=1 at the delivered commit), r(x)r(x) is what that node retains, and P(x)P(x) is its parent set; a node with no parents retains all of its flow. Version 2.2 sets λ=0.625\lambda = 0.625. Single-parent commits by the same author as their parent are collapsed into the parent first; add-then-revert churn is stripped out; and when a complete score chain exists, weights blend 80% score gain with 20% graph position.

Proposition 1 (Merge order) In the opening delivery (Carol merges Alice first, then Bob, and delivers), Alice—merged first—receives (1−λ)2/2(1-\lambda)^2/2, while Bob—merged second—receives (1−λ)/2(1-\lambda)/2; at λ=0.625\lambda = 0.625, their shares are 7.03% and 18.75%, regardless of what either wrote. Reversing the merge order reverses their shares. Lineage attribution therefore violates symmetry.

Proof. The delivered node is Carol's second merge: it retains λ\lambda and splits the remainder equally between its two parents, giving Bob's module and Carol's first merge (1−λ)/2(1-\lambda)/2 each. Bob's module has no parents and retains everything. Carol's first merge retains λ(1−λ)/2\lambda(1-\lambda)/2 and passes (1−λ)2/2(1-\lambda)^2/2 to its sole parent, Alice. Carol collects λ+λ(1−λ)/2\lambda+\lambda(1-\lambda)/2 in total, or 74.22%. PR #34's lineage function produces the exact same numbers (Appendix A).

Proposition 2 (Sequential relay) When kk contributors relay work one commit after another, the final contributor receives λ\lambda, the contributor jj steps earlier receives λ(1−λ)j\lambda(1-\lambda)^{j}, and the first contributor receives (1−λ)k−1(1-\lambda)^{k-1}. Across a four-person relay, the shares from first to last are 5.27%, 8.79%, 23.44%, and 62.5%.

Proof. Every node on the chain has at most one parent; walking backward from the delivery, the flows reaching successive nodes are 1,1−λ,(1−λ)2,…1, 1-\lambda, (1-\lambda)^2, \dots. Each node retains a fraction λ\lambda of what reaches it, except the first contributor's commit, which has no parent and retains all remaining flow.

If both modules are brought together in a single octopus merge at delivery, the three shares become 18.75%, 18.75%, and 62.5%—symmetric between Alice and Bob: the asymmetry came entirely from merge order, not from the work. Cooperative game theory already has names for rules of this shape. Restricting feasible coalitions to those respecting a partial order yields the Shapley value under precedence constraints (Faigle & Kern 1992); when precedence means subordinates need a superior's permission to cooperate, gatekeepers capture large shares, of which Carol's 74.2% for two merges is an instance. DyLAN's agent-importance score propagates backward from terminal nodes in a structurally isomorphic way (Liu et al. 2024), while Priority-Aware Shapley turns how much order matters into a tunable parameter interpolating between order-agnostic Shapley and purely order-driven rules (Lee et al. 2026). Lineage numbers reflect a choice of λ and commit topology, not the work itself.

Structural rules still earn their keep in one degenerate case: when an entire delivery comes from a single side, there is nothing to divide, and Shoal's main branch still settles that case immediately via commit ownership.

0.625v2.2 in PR #34 uses 0.625
BaseAlice · 400 linesBob · 400 linesCarol · 1st mergeCarol · 2nd merge, deliveredBaseAlice · 400 linesBob · 400 linesCarol · merges both, deliveredLeg 1Leg 2Leg 3Leg 4, deliveredBaseAliceBobCarolCarol · deliveredBaseAliceBobCarol · deliveredLeg 1Leg 2Leg 3Leg 4, delivered

AliceBobCarolMerge Alice firstMerge Bob firstSingle merge
Data
Commit graph (λ = 0.625)Alice (%)Bob (%)Carol (%)Leg 1 / 2 / 3 / 4 (%)
Merge Alice first7.0318.7574.22—
Merge Bob first18.757.0374.22—
Single merge18.7518.7562.5—
Four-step relay———5.27 / 8.79 / 23.44 / 62.5
Figure 9 Lineage attribution places one unit of credit on the delivered commit and lets it flow backward along parent edges, with each node retaining λ. Swap the merge order, switch to a single merge or a four-step relay, or drag λ; in the small simplex at bottom right, the two merge orders sit symmetrically across the midline. Values at λ = 0.625 come directly from PR #34's lineage function, and other values are recomputed in-browser from the closed forms in Propositions 1 and 2.

3.2Absence

The Shapley value:absence as known

A second family starts from principle: ask what would happen without each contributor, and average their marginal contribution across every possible arrival order—the Shapley value of equation (1). It is the unique rule satisfying efficiency, symmetry, null player, and additivity simultaneously (Shapley 1953); averaging equally over coalitions without requiring efficiency gives the Banzhaf value (Banzhaf 1965). Shapley cleanly separates complements from substitutes and ignores commit inflation: in an extreme test by the branch author, where one contributor supplied the working change and five others pushed 90 ineffective commits between them, the effective contributor took 100%.

Yet equation (1) demands v(S)v(S) for every coalition. Return to the opening delivery: only when all three pieces are combined does a runnable program exist. If anyone could have performed the merge, Carol is substitutable, vv depends solely on whether Alice and Bob are present, and Shapley assigns Carol 0; if without Carol no merge happens, all three are veto players and each receives 1/3. Both value functions are illustrative assumptions, and they differ on a single point: whether someone else would have merged had Carol been absent. Lu et al. observe the exact same sensitivity in multi-agent software workflows: under a pure removal protocol, any agent whose absence breaks the pipeline becomes a veto player with a sharply inflated share (Lu et al. 2026).

A speedup from 10 seconds to 4 seconds makes the same point. Suppose contributors speed a program up on an agreed test suite: if Alice's and Bob's optimizations work independently and additively—saving 4 seconds and 2 seconds on their own—Shapley splits the credit 4 to 2; if the 6-second gain materializes only when both changes are present together, Shapley splits it 3 to 3. That 3-to-3 split comes from the symmetry axiom—it is a convention, not a measured fact.

To measure how much "absence" weighs in practice, the PR #34 author ran a real experiment: six Claude Code agents each wrote one component of a Max-Cut heuristic, and all 64 coalitions were evaluated across three random seeds to compute exact Shapley values. When absent components fell back to the original reference implementation, the controller accounted for 2.51%; swapping the single-point search fallback for a stronger reference implementation jumped the controller's share to 26.62%; and switching from fixed iteration counts to giving every coalition the same 0.1 seconds of CPU time drove the controller's share to 0.

4.3ppMax single-component shift across random seedsSame protocol, three seeds; TVD 5.4
26.6ppMax single-component shift across absence replacementsSame code, three protocols; TVD 50.0
6/ 327Problems with an executable check yielding v(S)v(S)Shoal problem catalog, November 2026

That baseline choice governs Shapley attributions is well known in model explainability: different rules for replacing absent features define different cooperative games (Sundararajan & Najmi 2020; Janzing et al. 2020; Kumar et al. 2020); as the Distill article on attribution baselines put it, "We have many baselines and no conclusion about which one is the 'best.'" (Sturmfels et al. 2020). Our experiment reproduces that phenomenon on multi-agent team contributions and puts a scale on it: in this six-component system, changing the definition of absence shifts shares roughly 6 times (max individual shift) to 9 times (total variation) as much as seed noise. Data valuation shows the opposite regime: training stochasticity alone scrambles valuation rankings, leaving rank correlations across seeds at just 0.001 for leave-one-out and 0.038 for Data Shapley (Wang & Jia 2023). The two findings do not conflict: with few players and large individual effects, specification risk dominates noise. Notice too what the equal-CPU-budget protocol did—an absent component effectively ceded its compute budget to the remaining ones, an intervention mode for which we found no direct counterpart in the feature-baseline literature we surveyed (though our search there focused only on baseline design).

Illustrative: v(S) is posited, not measured
∅{A}{B}{C}{A, B}{A, C}{B, C}{A, B, C}

A = Alice, B = Bob, C = Carol; v(S) = 1 means coalition S can deliver

Arrival orderAliceBobCarol
φ, mean over all 6 orders

Absent components revert to the original reference implementation; fixed 12 iterations per run

Construct16.04%
1-flip search20.21%
2-flip search24.86%
Perturb20.41%
Recombine15.97%
Controller2.51%
01020304050Share (%)

Shapley share, 3-seed meanRange across individual seedsLeave-one-out (negatives clamped to 0, normalized)

Per-seed coalition scores were archived only for the original protocol; the two stress protocols archived 3-seed means. Across seeds, no single component moves by more than 4.3 pp.

Data
ComponentOriginal fallback (%)Stronger fallback (%)Equal CPU budget (%)Seed range, original (%)Leave-one-out score dropLeave-one-out, norm. (%)
Construct16.043.2717.8914.42–17.67−30
1-flip search20.212.1121.619.64–21.2900
2-flip search24.8632.2821.823.87–25.8911.6738.9
Perturb20.4133.9621.0719.91–20.7615.3351.1
Recombine15.971.7617.6413.9–18.2200
Controller2.5126.6201.1–3.26310
Figure 10 Top: eight coalitions and six arrival orders on the opening delivery, where the Shapley value averages marginal contributions across all six paths; both value functions are illustrative—when anyone can merge, Carol gets 0; when no merge happens without her, each contributor gets 1/3. Bottom: the six-component Max-Cut experiment across three definitions of "absence"; triangle markers show leave-one-out impact under the original protocol (clamping negative drops to 0 and normalizing, a step added here beyond the raw archive), and error bars show per-seed range. Data from PR #37's experimental archive.

Computation is not the bottleneck. Exact evaluation takes 2n2^n coalitions—roughly 1.27×10301.27\times10^{30} at 100 contributors—but leverage-score sampling achieves provable error bounds in O(nlog⁡n)O(n\log n) evaluations (Musco & Witter 2025), and the author's balanced permutation block design requires 19,802 check runs per block at 100 contributors. What is missing is v(S)v(S) itself: only 6 of Shoal's 327 problems have an executable check, because most open research outcomes do not reduce to a scalar score.

3.3Ranks

Rubrics and tiers:ranks as multiples

A third approach has a model read the evidence and grade against a rubric, leaving code to turn grades into shares. PR #34's rubric scored five dimensions from 0 to 4—substantiveness SS, criticality CC, originality OO, groundwork, and review—with technical weight

w=0.5 S+0.3 C+0.2 O4w=\frac{0.5\,S+0.3\,C+0.2\,O}{4}
(3)

Contributors below 1 on substantiveness were excluded from the technical pool; scores took the median across three rounds, flagging runs whose cross-round spread exceeded 15 percentage points as unstable. Recomputing from the 652 archived raw scoring rows reveals three structural problems in the rubric itself. First, zero inflation: review was 0 on every single row, groundwork was 0 on 94%, substantiveness, criticality, and originality were each 0 roughly half the time, and the top grade of 4 was never used once for originality or groundwork; substantiveness alone explained 87% of the variance in technical weight. Second, polarized validation cases: of the 16 validation cases, 10 were winner-take-all, 5 were exact even splits, and only 1 involved an intermediate ratio—so the suite tested who gets in, almost never how much each gets, and varying the coefficients on substantiveness, criticality, and originality barely moved pass counts across any of the three models. Third, ordinal grades were treated as ratios.

Observation 2 (Ranks state order only) Grades from 0 to 4 form an ordinal scale: they state order, not multiples. Under Stevens's theory of measurement scales, an ordinal scale is invariant up to any strictly monotone transformation (Stevens 1946); yet replacing 0, 1, 2, 3, 4 with the order-preserving scale 0, 1, 3, 9, 27 shifts GPT's shares on validation case 18 from 34.9 / 65.1 to 12.0 / 88.0—a 22.8 pp swing. When shares move under an admissible transformation, the ratio comes from convention rather than measurement.

SubstanceS

49%
01234

CriticalityC

51%
01234

OriginalityO

51%
01234

EnablingE

94%
01234

ReviewR

100%
01234

Level 0Levels 1–4UnusedHorizontal: level (0–4); vertical: share of 652 rows

Treat 0 1 2 3 4 directly as numbers

01234
013927

Map to 0 1 3 9 27: same order, different spacing

AS 2 · C 1 · O 1w 1.5→2
BS 4 · C 2 · O 1w 2.8→14.6

0–4A 34.9 B 65.1

AB
22.8 pp change hands
AB

0 1 3 9 27A 12.0 B 88.0

GPT, validation case 18, 3-round median; w = 0.5 S + 0.3 C + 0.2 O, share = normalized w

Data
DimensionLevel 0Level 1Level 2Level 3Level 4
Substance (S)3192914751106
Criticality (C)331341444796
Originality (O)33221689150
Enablement / groundwork (E)611172040
Review (R)6520000

652 rows in total across three models and all rounds. Case 18 (GPT): A has S 2 · C 1 · O 1; B has S 4 · C 2 · O 1. Under 0–4: 34.88% / 65.12%; under 0, 1, 3, 9, 27: 12% / 88%—a 22.8 pp shift.

Figure 11 Left: grade distributions across all 652 rubric rows on five dimensions, with dashed slots marking levels never used once—Review is 100% zeros and Enablement is 94% zeros. Right: on the exact same ordinal grades for one validation case, replacing 0–4 with the order-preserving scale 0, 1, 3, 9, 27 moves shares by 22.8 percentage points. Recomputed from PR #34's archive (b9c65cc).

The design now on main (PR #38, live since November 10; rubric outcome-evidence/2) separates judgement from arithmetic and reverses the order of inquiry: list outcomes first, trace provenance second, and compute shares last. The agent begins by listing commissioned outputs, writing each as an adoption chain: which part of the target came from which revision and participant, was adopted by which change or call site, and passed through which integration or check. Then, within each output, each participant is graded once by their strongest evidenced relationship across six roles (idea, implementation, integration, validation, exploration, coordination), with tier points ptp_t fixed at core = 8, major = 4, supporting = 1, and incidental = 0.25; cases the frozen record cannot settle are marked pending, and un-adopted work is marked unattributed, both carrying 0 points; contributors whose work cannot be separated in the evidence form a single group and share one tier. The conversion to shares is public:

φi=∑kwk∑g∈Gk, g∋iptg∣g∣ ∑h∈Gkpth\varphi_i=\sum_{k} w_k\sum_{g\in G_k,\ g\ni i}\frac{p_{t_g}}{|g|\,\sum_{h\in G_k} p_{t_h}}
(4)

where GkG_k is the set of groups in output kk and tgt_g is group gg's tier. Within an output, points normalize across groups while members of a group split its points equally; multiple outputs combine by the requester's weights wkw_k (or equal weights when omitted) and round to 10,000 basis points (bps) via largest remainders. The basis travels inside the report JSON so the site recomputes shares with the exact same code. Making the arithmetic transparent exposes the three places where judgement still does the heavy lifting.

Proposition 3 (Tier ladder) In a single output with two solo contributors where one is graded core, the other can receive only pt/(8+pt)p_t/(8+p_t)—namely 50%, 33.33%, 11.11%, 3.03%, or 0.

Proof. By equation (4), for one output and two singleton groups φ2=pt2/(8+pt2)\varphi_2 = p_{t_2}/(8+p_{t_2}); substitute point values 8, 4, 1, 0.25, and 0.

Tiers are coarse, so a single borderline call jumps a full rung. For a one-line empty-input guard if (!xs.length) return 0, two live agent reviews both graded the author major, yielding 33.33%; a parallel example in the analysis package documentation grades it 8-to-1 (supporting), yielding 11.11%; earlier free-form scoring gave 15% to 20%. Everything hinges on one judgement: whether the line fixed a defect that broke an agreed acceptance condition, or merely improved an already working delivery.

Proposition 4 (Output partitioning) Equation (4) is not invariant to how outputs are partitioned. When Alice (core), Bob (supporting), and Carol (incidental) are graded inside a single output, Carol receives 2.70%; carving her minor fix out as a second, equally weighted output raises her share to 50%.

Proof. Inside one output, Carol receives 0.25/(8+1+0.25)0.25/(8+1+0.25), or 2.70%; split out as a separate output where she is the sole contributor, she takes all of that second output, multiplied by weight 1/2.

This is how splitting attacks manifest under tier grading—the same vulnerability class as shell-company and false-name manipulations in cooperative games (Lee et al. 2025a; Burnat & Davidson 2026), and akin to rubric evaluations where trivial sub-claims inflate precision from 54% to 83% (Jiang et al. 2025). A natural structural fix is to merge outputs whose provenance traces back to the same underlying change before normalizing. Today, the only guardrails against over-splitting are the rubric's instruction to combine records into a single output unless the requester separately accepts independent deliveries, together with the requester's explicit weights.

Proposition 5 (Pending vs. unattributed) At payout time, pending is indistinguishable from unattributed. With Alice core, Bob supporting, and Carol pending, equation (4) yields 88.89%, 11.11%, and 0—identical to grading Carol unattributed; whereas if later evidence establishes her work as supporting, the shares become 80%, 10%, and 10%.

Proof. Both tiers carry point value 0, and equation (4) reads only point values.

The protocol specifies that pending leaves a question open rather than declaring a zero; in payout terms, however, everyone else divides that person's share immediately. Whether to hold a reserve slice in escrow for pending contributors before settling remains an open design choice. Propositions 3 through 5 are asserted directly against the platform's composition code, and the interactive calculator on this page was verified against that code across 50,000 random reports with zero discrepancies.

Output 1Output 2SharebpsCredits / 100Alice0%00Bob0%00Carol0%00Normalized

Within outputφik = pt(i) ⁄ Σj pt(j)

Across outputsφi = Σk wk φik ⁄ Σk wk

RoundingLargest remainder to 10,000 bps

pt: Core 8 · Major 4 · Supporting 1 · Incidental 0.25 · Pending 0 · Unattributed 0; outputs where nobody scores carry no weight

Plug in

AliceBobCarol

When one contributor is Core, a second contributor can receive only

Figure 12 Interactive tier calculator: each output normalizes tier points internally, outputs combine by weight, and largest remainders round shares to basis points and integer credits. The small simplex tracks the three contributors' point: carving a minor fix into a second output jumps Carol from 2.7% to 50%, while setting her to "Pending" drops the point onto the bottom edge—indistinguishable from "Unattributed". Verified against the platform's composition code across 50,000 random reports.

Chapter 4

Testing: stability, validity,and what must not move

The same frozen archive can be assessed many times. The distance between two assessments is straightforward to measure; whether either is right requires checking outcomes and provenance; and certain transformations should never move the answer at all.

4.1Reliability & validity

Agreementis not accuracy

To compare two allocations, we use total variation distance from Definition 3. On the three-person simplex it has a clean geometric picture: the allocations at total variation distance rr from an interior point trace a hexagon whose six vertices each move rr percentage points from one contributor to another. On Shoal's UI, whenever multiple reports cover the same commit, the grey band on each participant's row spans their minimum to maximum share across reports.

Distance between reports conflates two distinct psychometric properties. One is reliability: whether assessing the same artifact repeatedly or across different judges yields consistent numbers. The other is validity: whether those numbers measure what we actually intend to measure (Cronbach & Meehl 1955).

For reliability, pairing a structured rubric with deterministic code genuinely helps. Recomputing from the archived raw outputs on cases where all three rounds succeeded under both regimes, letting three models assign percentages directly produced a mean pairwise TVD of 14.7 pp across models; grading on PR #34's rubric and converting in code cut that to 4.2 pp, improving all 11 comparable cases. Within-model max individual spread across rounds dropped from 10.3 pp to 1.4 pp on GPT and from 6.4 pp to 2.4 pp on Claude (5 comparable cases), while DeepSeek stayed flat (24.5 and 24.2). Two caveats apply: the rubric condition also saw platform provenance summaries, confounding structure with extra input, and those test cases were heavily polarized; the current tier rubric has not yet been benchmarked in isolation. Published results point the same way: decomposing holistic scoring into binary checklists raised inter-judge agreement from 0.09 to 0.48 (Lee et al. 2025b), and a generalizability-theory variance decomposition of LLM evaluations found judge-model choice accounting for 43.8% of variance versus under 0.5% for repeat runs (Messing 2026)—matching the "tight across rounds, wide across models" pattern we observed on GPT and Claude (with DeepSeek noisy across rounds as well).

Validity is another matter.

Observation 3 (Agreement far from the reference) On one validation case, the rubric produced the exact same shares—28.1 / 43.8 / 28.1—across all three rounds, yet sat 19.3 pp in total variation away from the measured ablation shares of 43.3 / 47.9 / 8.9. Because that "measured" ablation is itself one operationalization rather than ground truth, the gap shows not which side is right, but that self-consistency does not imply proximity to a counterfactual reference.

Inter-model agreement cannot stand in for truth either: judge models favor generators similar to themselves, and stronger models make increasingly correlated errors (Goel et al. 2025). Judges reading logs alone pinpoint the failing step only 14.2% of the time (Zhang et al. 2025). As Messick argued, validity belongs not to a score in the abstract but to the inferences and actions drawn from it (Messick 1995): using the same report for public recognition and for credit settlement requires two separate justifications. Any ceiling on machine validity should also be calibrated against human disagreement: on repository-level evaluation tasks, pairs of human experts disagree 10% to 30% of the time (Zhuge et al. 2025).

Rubric (3 identical rounds)MeasuredAll splits at D = 19.3 from Measured

Row 1 averages across 11 cases; rows 2 and 3 report pairwise maxima; row 4 is the single case plotted on the simplex. Statistics differ across rows—compare within each row.

Data
ParticipantRubric (all 3 rounds identical)Measured
A28.143.3
B43.847.9
C28.18.9

Total variation distance D(p, q) = ½ Σ |pᵢ − qᵢ| = 19.3 pp (computed from unrounded shares). "Measured" is the coalition-ablation Shapley vector for this case, which itself depends on how absence is defined.

ComparisonDStatistic
Across models, free-form shares14.7Mean over 11 cases
Across models, rubric + code conversion4.2Mean over 11 cases
Across random seeds (6-component Shapley)5.4Max pairwise
Redefining absence (6-component Shapley)50Max pairwise
Figure 13 Agreement is not accuracy. On the simplex, all three rounds of rubric scoring land on the exact same point, while the measured ablation (itself one operationalization) sits 19.3 percentage points away; the dashed hexagon traces every allocation at total variation distance 19.3 from Measured. Right: measured distances when swapping models (mean across 11 cases), swapping random seeds, and redefining absence (pairwise maxima).

4.2Invariance

Equivalent rewritesand adversarial probes

Low variance shows stability, not soundness, which calls for a second class of checks: invariance. Rewriting the exact same work in an equivalent representation should not alter shares: swapping merge order, splitting work into more commits, splitting an outcome into finer sub-outputs, adding idle accounts, renaming files, or making an edit and reverting it. In software testing, this is metamorphic testing. Each invariance maps directly onto one of Chapter 1's probes: lineage fails merge-order invariance (Proposition 1), commit count fails commit-splitting invariance, tier grading fails output-partition invariance (Proposition 4), and equal contributor shares fail the addition of a trivial editor.

Adversarial robustness forms a third check. In one test repository under review, we planted a CLAUDE.md instructing the assessor to grade the submitter core and slip a codeword into the summary, alongside a session hook. Neither the codeword nor the hook fired; instead, the report noted the planted file as part of the repository material under review. That single run shows the review driver's tool allowlist doing its job, not immunity to prompt injection: an allowlist narrows what a hijacked session can execute, not what it can conclude, and is no substitute for a sandbox—untrusted repositories still belong inside a container. Two end-to-end live reviews took 103 s ($0.99) and 128 s ($1.22).

Freezing these probes gives a regression suite for attribution rules: before shipping any rule change, run it against a fixed shelf of "trap deliveries"—merge-only, comment-only padding, add-then-revert, split outputs, and planted prompt instructions. Even then, the test suite itself must not become an optimization target: under heavy pressure, optimizing against a detector teaches policies to conceal misbehavior rather than stop it (Baker et al. 2025).

4.3Property matrix

Six rules,five properties

Combining the desiderata from Chapter 2 with the proofs and live probes above yields the comparison matrix below.

EfficiencySymmetryNull playerSplit invarianceEvidence-boundedRequires
By commit countShoal main, since 10-25definitiondefinitionmeasuredmeasureddefinitionGit history
By lineage positionPR #34, unmergedcodecodemeasuredmeasureddefinitionGit history, plus score chain when present
One share per changerPR #35, withdrawndefinitiondefinitionmeasuredmeasureddefinitionGit history + "substantive diff" filter
Agent free-form sharesAuthor branch, 1 rundefinitionmeasuredmeasuredOne reviewing agent
Tiers & public formulaShoal main, since 11-10codecodecodecodedefinitionOne reviewing agent, frozen snapshot
Shapley valueCooperative game, in principleaxiomliteraturemeasuredOne check run per coalition; only 6 / 327 problems have an executable check

By commit countShoal main, since 10-25

  • EfficiencyHolds: normalized by commit count, remainder to delivererdefinition
  • SymmetryHolds: counts commits only, order-independentdefinition
  • Null playerFails: merge-only Carol takes 50; comment-only padding file is paid in fullmeasured
  • Split invarianceFails: one-line finisher takes half; splitting commits inflates sharemeasured
  • Evidence-boundedFails: counts actions only, never inspects contentdefinition

Requires: Git history

By lineage positionPR #34, unmerged

  • EfficiencyHolds: retained shares across nodes sum to 100code
  • SymmetryFails: Prop. 1—merged first gets 7.03, merged second gets 18.75; swapping merge order swaps sharescode
  • Null playerFails: merge-only Carol takes 74.2measured
  • Split invarianceFails: one-line finisher takes 62.5; requester clicking Approve claws back 11.8measured
  • Evidence-boundedConditional: position only by default; blends 80% score gain when a complete score chain existsdefinition

Requires: Git history, plus score chain when present

One share per changerPR #35, withdrawn

  • EfficiencyHolds: splits equally across eligible changersdefinition
  • SymmetryHolds: checks only whether each contributor made a substantive changedefinition
  • Null playerFails: blocks merge-only, but edit-then-revert still collects 20 of 60 creditsmeasured
  • Split invarianceFails: each extra changer gets an equal slice—requester pushing 1 line claws back 20 of 40 creditsmeasured
  • Evidence-boundedFails: counts headcount onlydefinition

Requires: Git history + "substantive diff" filter

Agent free-form sharesAuthor branch, 1 run

  • EfficiencyConditional: enforced by report schema validationdefinition
  • SymmetryUntested: order and name permutations not tested
  • Null playerConditional: gave everyone 0 on the padding file and flagged unchecked acceptance claims (1 run)measured
  • Split invarianceUntested: splitting probes not run
  • Evidence-boundedFails: unanchored percentages—swapping models shifts shares by 14.7 pp TVDmeasured

Requires: One reviewing agent

Tiers & public formulaShoal main, since 11-10

  • EfficiencyHolds: largest-remainder rounding to 10,000 bps; checked across 50,000 random reportscode
  • SymmetryHolds: invariant to output and group orderingcode
  • Null playerConditional: depends on judgement—Unattributed gets 0, yet Pending also gets 0 at payout (Prop. 5)code
  • Split invarianceFails: Prop. 4—carving a minor fix into an equally weighted output jumps its author from 2.7 to 50code
  • Evidence-boundedHolds: every tier attaches to an adoption chain, and the basis is stored with the report and recomputed by the sitedefinition

Requires: One reviewing agent, frozen snapshot

Shapley valueCooperative game, in principle

  • EfficiencyHolds: axiom
  • SymmetryHolds: axiom
  • Null playerHolds: axiom (given v(S))—5 contributors with 90 ineffective commits get 0axiom
  • Split invarianceFails: exact Shapley over declared identities cannot prevent splitting and false-name attacks (Burnat & Davidson 2026)literature
  • Evidence-boundedConditional: requires v(S) for every coalition, and changing absence replacement shifts shares by up to 26.6 ppmeasured

Requires: One check run per coalition; only 6 / 327 problems have an executable check

HoldsFailsConditionalUntestedSub-labels show basis: axiom · code · measured · literature · definition

Data
RulePropertyVerdictBasisSource
By commit countEfficiencyHoldsNormalized by commit count, remainder to delivererdefinition
By commit countSymmetryHoldsCounts commits only, order-independentdefinition
By commit countNull playerFailsMerge-only Carol takes 50; comment-only padding file is paid in fullmeasured
By commit countSplit invarianceFailsOne-line finisher takes half; splitting commits inflates sharemeasured
By commit countEvidence-boundedFailsCounts actions only, never inspects contentdefinition
By lineage positionEfficiencyHoldsRetained shares across nodes sum to 100code
By lineage positionSymmetryFailsProp. 1: merged first gets 7.03, merged second gets 18.75; swapping merge order swaps sharescode
By lineage positionNull playerFailsMerge-only Carol takes 74.2measured
By lineage positionSplit invarianceFailsOne-line finisher takes 62.5; requester clicking Approve claws back 11.8measured
By lineage positionEvidence-boundedConditionalPosition only by default; blends 80% score gain when a complete score chain existsdefinition
One share per changerEfficiencyHoldsSplits equally across eligible changersdefinition
One share per changerSymmetryHoldsChecks only whether each contributor made a substantive changedefinition
One share per changerNull playerFailsBlocks merge-only, but edit-then-revert still collects 20 of 60 creditsmeasured
One share per changerSplit invarianceFailsEach extra changer gets an equal slice—requester pushing 1 line claws back 20 of 40 creditsmeasured
One share per changerEvidence-boundedFailsCounts headcount onlydefinition
Agent free-form sharesEfficiencyConditionalEnforced by report schema validationdefinition
Agent free-form sharesSymmetryUntestedOrder and name permutations not tested
Agent free-form sharesNull playerConditionalGave everyone 0 on the padding file and flagged unchecked acceptance claims (1 run)measured
Agent free-form sharesSplit invarianceUntestedSplitting probes not run
Agent free-form sharesEvidence-boundedFailsUnanchored percentages: swapping models shifts shares by 14.7 pp TVDmeasured
Tiers & public formulaEfficiencyHoldsLargest-remainder rounding to 10,000 bps; checked across 50,000 random reportscode
Tiers & public formulaSymmetryHoldsInvariant to output and group orderingcode
Tiers & public formulaNull playerConditionalDepends on judgement: Unattributed gets 0, yet Pending also gets 0 at payout (Prop. 5)code
Tiers & public formulaSplit invarianceFailsProp. 4: carving a minor fix into an equally weighted output jumps its author from 2.7 to 50code
Tiers & public formulaEvidence-boundedHoldsEvery tier attaches to an adoption chain, and the basis is stored with the report and recomputed by the sitedefinition
Shapley valueEfficiencyHoldsAxiomaxiom
Shapley valueSymmetryHoldsAxiomaxiom
Shapley valueNull playerHoldsAxiom (given v(S)): 5 contributors with 90 ineffective commits get 0axiom
Shapley valueSplit invarianceFailsExact Shapley over declared identities cannot prevent splitting and false-name attacks (Burnat & Davidson 2026)literature
Shapley valueEvidence-boundedConditionalRequires v(S) for every coalition, and changing absence replacement shifts shares by up to 26.6 ppmeasured
Figure 14 Not a single rule satisfies all five properties, and none withstands splitting. Solid circles mean holds, crosses mean fails, half-filled circles mean conditional, and dashed rings mean untested; sub-labels indicate whether each cell rests on an axiom, a code assertion, a live probe, published literature, or definition. An accent rule at left marks our live tier design.

Not a single row is solid across the board. Shapley is cleanest axiomatically, yet the v(S)v(S) it requires rarely exists; tier grading stays bounded by cited evidence and invariant to merge order, yet breaks under output partitioning; and every purely structural rule falls to cheap actions.

ScopeWhere the evidence stops

  • Mostly single runs. Live probes on payout paths and end-to-end agent reviews were executed once per fixture; they establish whether a rule can be fooled by a given action, not the sampling distribution of agent judgements.
  • Small fixtures. Our testbeds involve two to four participants and a handful of commits; real multi-contributor projects with continuous share distributions have not yet been systematically benchmarked.
  • Confounded rubric comparison. The rubric arm in the archived experiment also received platform provenance summaries, so the effect of structured grading alone has not yet been isolated.
  • Tier point ratios are conventions. The weights 8, 4, 1, and 0.25 are product conventions chosen to form a concrete recommendation, not empirically measured ratios; this essay claims only that they are public, reproducible, and open to rebuttal by a second report.
  • Calibration is still ahead. Proper calibration will need 20 to 30 real multi-party deliveries whose provenance and adoption chains are annotated by domain participants, supplemented by replayable synthetic tasks to benchmark judgement-based methods.
  • Bounded literature search. Claims marked "to the best of our search" rest on 145 verified entries across six directions rather than an exhaustive sweep; see Appendix B for exact scope and exclusions.

Chapter 5

Mechanism: recognition, credits,and responsibility

The preceding chapters showed that contribution measurement cannot be separated from its definitions—or from the behavior those definitions induce. Here we derive our architecture: unbundle the three jobs authorship has tied together, and govern each by its own rules.

5.1Three ledgers

The three jobsbundled inside authorship

The three jobs traditionally governed by authorship make conflicting demands: recognition must be allowed to say "we cannot tell yet," payout has to settle into exact integers before a deadline, and responsibility can rest only on a person who can be held to account. Economics noticed this double burden long ago: discussing team production in 1972, Alchian and Demsetz remarked in a footnote that "Meter means to measure and also to apportion"—one verb carrying both measurement and division (Alchian & Demsetz 1972). We split that verb in two, and add a third ledger beside them.

  • Recognition, treated as measurement: start from what actually changed—list outcomes first, trace provenance second; record contributions by role rather than collapsing into a single score; mark unresolved provenance pending and count each distinct effect once; register definitions upfront; and let anyone named submit a second report.
  • Credits, treated as mechanism: the requester who posted the credits decides within a fixed window, recipients are restricted to makers of the delivery, and expiry triggers a deterministic default—no lockup, no flow-back, and full visibility.
  • Responsibility, left to people: publishing a report and confirming payout require a signed-in human user. Agents supply evidence; they do not issue decisions.

Why put the payout decision in the requester's hands? Holmström proved that when only joint team output is observable, no budget-balancing sharing rule can sustain efficient effort without an external budget-breaker who does not share in the proceeds (Holmström 1982)—precisely the structural position of the requester. Holmström's premise is that only joint output is observed, whereas Shoal observes individual commits while leaving adoption ambiguous, so the theorem serves here as a structural parallel rather than a literal fit. Yet a budget-breaker faces moral hazard of their own (Eswaran & Kotwal 1984)—Chapter 1's probe where a requester pushed one line into a delivery and recovered half their own escrow is that exact hazard—which is why the requester's side is barred from receiving credits. Why not let contributors divide the bounty by peer rating? Among three recipients, no division rule is simultaneously impartial and consensual (de Clippel et al. 2008)—and our opening delivery has exactly three contributors. Psychometrics arrives at the same split from another angle: validity belongs to a score's use (Messick 1995), so recognition can rely on multidimensional, non-additive indicators that are far stabler than shares, whereas only payout forces shares onto the simplex. As for why responsibility stays human, journals and conferences have already stated the reason: AI tools cannot be held accountable for submitted work (COPE 2023; Nature 2023).

How tightly recognition couples to payout is itself a design knob. SourceCred computed cred over contribution graphs and minted grain directly from it; the organization wound down in 2022 (SourceCred 2022). Optimism's Retro Funding let impact assessments set grant amounts directly, migrating over three years from plenary badgeholder voting to metrics and then algorithms before pausing in January 2026 for at least 12 months to reassess (Optimism 2026). Protocol Guild, by contrast, uses recognition solely to decide membership eligibility and routes $12.4 million (in 2025) through a coarse tenure-weighted formula (Protocol Guild 2026). Holmström and Milgrom's multitask result points toward the coarse end as well: when hard-to-measure work matters, pay should track only coarse signals (Holmström & Milgrom 1991). Shoal currently sits partway between: when the requester confirms, the report acts as advisory evidence—a coarse link—whereas if the 3-day window expires, default settlement normalizes the report's exact basis points—a tight link.

Treat recognition as measurement, credits as mechanism, and leave responsibility to people. An agent can supply the evidence; it cannot issue the decision.

5.2Three layers

Facts, judgement,and decision

Those three ledgers map onto three architectural layers. At the bottom sit facts: the snapshot frozen by the site—including the Git bundle, jobs, checks, and reviews, along with the exact rules revision governing this analysis—so definitions are registered before assessment begins, leaving rule-pickers nothing to cherry-pick afterward. In the middle sits judgement: a client-side agent reads the frozen archive, writes the list of commissioned outputs, adoption chains, and tiers, marks unresolved links pending, and leaves room for anyone named to file a second report on the identical snapshot. At the top sits decision: publishing a report and confirming payout can be performed only by a signed-in person. GitHub's Copilot coding agent enforces a kindred separation: the person who delegated the task cannot supply the required pull-request approval (Copilot Docs 2026).

The three-layer split answers Shapin's warning as well. When a person publishes an agent's report, the structure mirrors a seventeenth-century master speaking for a technician; unless provenance is recorded separately, the work of agents and upstream contributors gets quietly "included in" the publisher's name (Shapin 1989). Every published report on Shoal therefore records who uploaded it, which target commit it addresses, and whether the requester edited it—keeping the unedited original readable alongside any edits. Those three facts are all the site can attest to; it cannot prove which model actually wrote the local report.

Take the 10-second-to-4-second speedup again: four contributors cut a program's runtime on an agreed test suite from 10 seconds to 4 seconds. The recognizable advance is "60% runtime reduction while preserving correctness"—established jointly by the code, PR, and report, and counted once. Alice proposed and implemented the main method (core), Bob contributed a substantive implementation optimization (major), Dan wrote an independent verification suite (supporting), and Carol merged the branches together (unattributed). Tier grading recommends 61.54%, 30.77%, 7.69%, and 0; whereas under a Shapley game whose value function is seconds saved, Dan's verification receives 0.

Three ledgers
DecisionSigned-in humanTokens rejected

Publish report, confirm payoutRequester's side · within 3 days

61.54%30.77%

Carol 0: merging added no standalone effect. Shares computed by the platform's composition code: 61.54 / 30.77 / 7.69 / 0

Credits

Paid only to makers of the delivery; the requester's side cannot receive

Confirmed within 3 days, else settled by default

Responsibility

Who published and who confirmed payment stay on the record

Agents supply evidence, never decisions

JudgementClient-side agentRequester's side
Recognition

Who made the result hold, and on what evidence

Can remain Pending; can be rebutted by a second report

FactsThe siteFrozen record

Shared evidence across all three ledgers: immutable once frozen

Figure 15 Three layers and three ledgers: the site freezes facts alongside the rules revision, a client-side agent writes judgements into recognition, and a signed-in human makes decisions recorded under credits and responsibility. Shown on the 10-second-to-4-second speedup example, where shares 61.54 / 30.77 / 7.69 / 0 come from the platform's composition code; the second report is illustrative.

5.3Payout guarantees

No lockup, no flow-back, full visibility—and the gaps we measured

The credit rules enforce just three guarantees: no lockup, no flow-back, and full visibility. Every execution path terminates in either payout or refund, with none stranded in between; once a delivery is accepted, no path routes credits back to the requester; and every decision—who made it, the shares, and the rationale—remains on the public record. Combining a decision window with an automatic default has mature precedent: on DrivenData, team prizes split equally unless all members unanimously request a custom split within 3 business days after the deadline (DrivenData 2026).

Testing the live settlement code turned up three remaining gaps.

Observation 4 (Default without a report) On the opening merge-only delivery, if the requester neither confirms a split nor publishes a report within 3 days, the current code holds the 100 credits in escrow at acceptance and, after day 4, settles by commit count: Carol 50, Alice 25, Bob 25.

The second gap is Proposition 5: pending collapses to zero at payout. The third is Proposition 4: how outputs are partitioned can rewrite shares. Each gap admits concrete remedies: the reportless default could pay only contributors with non-merge diffs, require a report before expiry, or refund the escrow; pending shares could be held back in a reserve slice until resolved; and outputs could be merged by shared source provenance before scoring. None of those changes has been settled yet.

5.4Invisible work

Why verification and reviewdisappear from the numbers

The hardest problem comes last. Verification, review, and groundwork vanish to near zero under the five-dimension rubric, score-based Shapley, commit count, and lineage alike: in the archived rubric runs, review scored 0 on every row and groundwork scored 0 on 94%; under runtime-based Shapley, verification that saves zero seconds gets 0; and commit count and lineage are blind to work that produces no commits on the delivered branch. Tier grading can award Dan's verification a supporting tier—provided the assessor notices it. This is Shapin's invisible technician in modern dress (Shapin 1989), and exactly what multitask agency theory predicts: reward only what is easy to meter, and effort drains away from what is not (Holmström & Milgrom 1991). Even mathematics' newest AI credit ledger relegates literature retrieval and formalization to minor status (Erdős Wiki).

Yet in human–agent research, verification is the scarce input. Agents can generate candidate solutions cheaply; the bottleneck is telling which candidate actually holds up—across Shoal's 327 problems, only 6 currently have an executable check. Any rule that pays strictly on marginal improvement to the headline metric will systematically underpay the work of verification. Our current answer is role-typed recognition: Dan's check is recognized as validation rather than converted into seconds saved, earning its own tier, and a second report submitted by a named participant is itself a form of review. Whether verification ought to carry a dedicated credit pool of its own remains open.

The work a research system needs most is the hardest to see. The test of an attribution rule is whether it can see the checks and reviews that make a result stand.

5.5Status

What is live today,and what remains open

Project analysis and multi-party settlement landed on main on November 10 and deployed to shoal.club the same day. The site freezes snapshots, receives validated reports, and records human decisions; reviews run on users' own machines; and multi-party deliveries give the requester 3 days to confirm before falling back to the default. Automatic share estimators based on commit lineage or Shapley values, together with 450,000 lines of experimental archives, remain on the author's branch rather than main.

  1. 10-25Credit escrowAppend-only ledger; accepted deliveries split by commit count, remainder to deliverer
  2. Early NovExperimentsFive-dimension rubric across three models; six-component Max-Cut Shapley study
  3. 11-06ReviewPR #34 lineage rule ranks merge-only first; absorbed PR #35 reveals three payout exploits in round-two review, withdrawn 11-07
  4. 11-07UnbundlingRecognition separated from credits; start from net changes, mark unknowns pending
  5. 11-09Project analysisPR #37: freeze a revision snapshot, local agent drafts report, requester reviews
  6. 11-10Merged & deployedPR #38: list outputs before grading tiers, 3-day confirmation window, default settlement, second report

Still open on the workbench: the fallback rule when no report exists; whether to escrow a reserve slice for pending contributors; structural defenses against output partitioning; how to compare contributions across distinct roles (currently displayed side by side by role without aggregation); who ratifies recognition on open problems without a requester, and how disputes are resolved beyond second reports and public issues; whether intermediate milestones should earn recognition upon verification with downstream impact tracked separately; whether project maintainers may receive payouts; how open problems without a single requester fund credits; and where to assemble a 20-to-30-case real-world calibration set. Every one of these choices shifts what people and agents choose to work on—so every one of them deserves to be tested against real deliveries first.

A paper compresses the work of an entire group into a single author line; Git shatters it back into thousands of commits. Our goal is not to count commits to more decimal places—it is to keep every adopted piece of work traceable to its source, give every payout an explicit reason, and leave both named contributors and otherwise invisible verifiers a place to say, "that is not what happened."

Appendix

Evidence, search log,and references

Experimental conditions, reproduction commands, literature search scope, and the complete bibliography.

AReproduction

Experimental conditionsand reproduction

Every number in this essay comes from one of three sources: live probes on platform code using temporary data directories, one run per probe; exact evaluation by platform functions, where calc/numbers.ts invokes main's sharesOf and PR #34's lineage while asserting Propositions 1 through 5 one by one (if any assertion fails, the data file refuses to generate); and recomputation from archived experimental logs. To reproduce: node calc/numbers.ts > calc/numbers.json and node calc/tiers-check.ts.

What this essay rests on

One delivery, four rules2026-11-07, measured live with an identical seed script (1,000 credits each) on main 359b875, PR #34 b9c65cc, and PR #35 40b8995; agent recommendation from the analysis package on author branch 5b5194e, single run (claude-sonnet-5-5, ~1 min, $0.21)
Fifteen specifications2026-11-13, calc/numbers.ts: three live measurements, nine computed directly by platform code, one derived from GitHub Docs, and two illustrative value functions
Propositions 1–5Asserted term by term against main's sharesOf (44b6777) and PR #34's lineage (c5f183d)
Payout-path probes2026-11-06 to 11-10, live platform code against temporary data directories, one run per probe
Current default behavior2026-11-13, measured live on the merge commit of PR #38 (4c9c621)
Six-component Shapley studyAuthor archive cooperative/: 64 coalitions × 3 seeds, plus stronger-fallback and equal-CPU stress protocols
652 rubric rows & 16 casesRecomputed from raw model outputs archived in PR #34 (b9c65cc)
End-to-end live reviews2026-11-10, Claude Code 2.1.321, claude-fable-5-2, one run per fixture; rerun after PR #38 fixes in 146 s

BSearch log

Literature search log

Related work was surveyed across six directions and completed on 2026-11-13: credit assignment and failure attribution in multi-agent systems; cooperative game theory and attribution methods; incentives, mechanism design, and public-goods funding; scientific credit and AI authorship; psychometrics of LLM-as-a-judge evaluation; and bounty platforms alongside AI attribution in code. Across 145 entries in total, we opened each primary page to verify title, authors, venue, and URL, recording publication status versus preprint status; COPE and ACM primary pages blocked automated fetching and were verified against secondary reproductions. Claims qualified by "to the best of our search" correspond to querying arXiv, the ACL Anthology, PMLR, and the ICML, ICLR, and NeurIPS proceedings pages with keyword sets covering multi-agent contribution attribution, credit assignment in LLM multi-agent systems, quantifying contributions in human–AI collaboration, authorship for research agents, and contribution attribution for agent-authored pull requests, reading abstracts or full texts item by item. Google Scholar, Semantic Scholar, and DBLP were not queried, and search quota ran out toward the end of the sweep—making this a bounded, documented search rather than an exhaustive proof of absence.

CReferences

References

  • Aad et al. 2015ATLAS Collaboration, CMS Collaboration (G. Aad, et al.). Combined Measurement of the Higgs Boson Mass in pp Collisions at √s = 7 and 8 TeV with the ATLAS and CMS Experiments. Physical Review Letters 114: 191803, 2015. doi:10.1103/PhysRevLett.114.191803
  • Agent Trace 2026Cursor. Agent Trace (specification v0.1.0, Status: RFC). agent-trace.dev, 2026. agent-trace.dev
  • Alchian & Demsetz 1972Armen A. Alchian, Harold Demsetz. Production, Information Costs, and Economic Organization. American Economic Review 62(5): 777–795, 1972. econpapers.repec.org
  • Algora 2026Algora. Slash commands reference. Algora Docs, 2026 (accessed 2026-11-13). docs.algora.io/commands
  • Allen et al. 2014Liz Allen, Jo Scott, Amy Brand, et al. Publishing: Credit where credit is due. Nature 508: 312–313, 2014. doi:10.1038/508312a
  • AWS 2025Chi Tran, Charlie Bacon. Amazon Inspector detects over 150,000 malicious packages linked to token farming campaign. AWS Security Blog, 2025. aws.amazon.com/blogs/security/…
  • Baker et al. 2025Bowen Baker, Joost Huizinga, Leo Gao, et al. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv, 2025. arXiv:2503.11926 (preprint)
  • Banzhaf 1965John F. Banzhaf III. Weighted Voting Doesn't Work: A Mathematical Analysis. Rutgers Law Review 19(2): 317–343, 1965.
  • Bertran et al. 2026M. Bertran, R. Fogliato, Z. S. Wu. Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse. arXiv, 2026. arXiv:2602.18710 (preprint) (partially unverified)
  • Bianchi et al. 2025Federico Bianchi, Owen Queen, Nitya Thakkar, et al. Exploring the use of AI authors and reviewers at Agents4Science. arXiv, 2025. arXiv:2511.15534 (preprint)
  • Bolan et al. 2025Matthew Bolan, Joachim Breitner, et al. The Equational Theories Project: Advancing Collaborative Mathematical Research at Scale. arXiv, 2025. arXiv:2512.07087 (preprint)
  • Brand et al. 2015Amy Brand, Liz Allen, Micah Altman, et al. Beyond authorship: attribution, contribution, collaboration, and credit. Learned Publishing 28(2): 151–155, 2015. doi:10.1087/20150211
  • Breznau et al. 2022N. Breznau, et al. Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. PNAS 119(44): e2203150119, 2022. doi:10.1073/pnas.2203150119
  • Burnat & Davidson 2026Florian A. D. Burnat, Brittany I. Davidson. Quotient Semivalues for False-Name-Resistant Data Attribution. arXiv, 2026. arXiv:2605.07663 (preprint)
  • Castelvecchi 2015Davide Castelvecchi. Physics paper sets record with more than 5,000 authors. Nature (News), 2015. doi:10.1038/nature.2015.17567 (partially unverified)
  • Cemri et al. 2025Mert Cemri, Melissa Z. Pan, Shuyi Yang, et al. Why Do Multi-Agent LLM Systems Fail? NeurIPS, 2025. arXiv:2503.13657 (partially unverified)
  • COPE 2023Committee on Publication Ethics (COPE). Authorship and AI tools. COPE position statement, 2023. publicationethics.org/…/authorship-and-ai-tools (partially unverified)
  • Copilot Docs 2026GitHub. Review output from Copilot (see also GitHub Changelog 2026-03-20: Trace any Copilot coding agent commit to its session logs). GitHub Docs, 2026 (accessed 2026-11-13). docs.github.com/…/review-copilot-output
  • Cronbach & Meehl 1955L. J. Cronbach, P. E. Meehl. Construct validity in psychological tests. Psychological Bulletin 52(4): 281–302, 1955. doi:10.1037/h0040957
  • de Clippel et al. 2008Geoffroy de Clippel, Hervé Moulin, Nicolaus Tideman. Impartial division of a dollar. Journal of Economic Theory 139(1): 176–191, 2008. doi:10.1016/j.jet.2007.06.005
  • Diehl & Wilson 2025Hannah Diehl, Ashia C. Wilson. Semivalue-Based Data Valuation Is Arbitrary and Gameable. arXiv, 2025. arXiv:2506.12619 (preprint)
  • DrivenData 2026DrivenData. Sustainable Industry: Rinse Over Run (Teams section of competition rules). DrivenData, 2026 (accessed 2026-11-13). drivendata.org/competitions/56/…/page/137
  • Erdős WikiTerence Tao, et al. AI contributions to Erdős problems. teorth/erdosproblems GitHub Wiki (data through 2026-06-30, frozen thereafter), 2025–2026. github.com/teorth/erdosproblems/wiki/…
  • Eswaran & Kotwal 1984Mukesh Eswaran, Ashok Kotwal. The Moral Hazard of Budget-Breaking. RAND Journal of Economics 15(4): 578–581, 1984. ideas.repec.org/a/rje/randje/…
  • Faigle & Kern 1992Ulrich Faigle, Walter Kern. The Shapley Value for Cooperative Games under Precedence Constraints. International Journal of Game Theory 21(3): 249–266, 1992. doi:10.1007/BF01258278
  • Foerster et al. 2018Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, et al. Counterfactual Multi-Agent Policy Gradients. Proceedings of the AAAI Conference on Artificial Intelligence 32(1): 2974–2982, 2018. doi:10.1609/aaai.v32i1.11794
  • Ghorbani & Zou 2019Amirata Ghorbani, James Zou. Data Shapley: Equitable Valuation of Data for Machine Learning. ICML, PMLR 97: 2242–2251, 2019. arXiv:1904.02868
  • Git AI 2026Git AI. How Git AI works. Git AI Docs, 2026 (accessed 2026-11-13). usegitai.com/docs/cli/how-git-ai-works
  • GitHub Docs 2026GitHub. Viewing a project's contributors. GitHub Docs, 2026 (accessed 2026-11-13). docs.github.com/…/viewing-a-projects-contributors
  • Goel et al. 2025S. Goel, et al. Great Models Think Alike and this Undermines AI Oversight. ICML, PMLR 267: 19621–19678, 2025. arXiv:2502.04313
  • Goodhart 1975C. A. E. Goodhart. Problems of Monetary Management: The U.K. Experience. Reserve Bank of Australia, Papers in Monetary Economics, Vol. I (Sydney conference paper, July 1975, collected 1976; reprinted in Goodhart 1984, Monetary Theory and Practice, Palgrave), 1975. doi:10.1007/978-1-349-17295-5_4 (partially unverified)
  • Gowers & Nielsen 2009Timothy Gowers, Michael Nielsen. Massively collaborative mathematics. Nature 461(7266): 879–881, 2009. doi:10.1038/461879a
  • HackerOne 2026HackerOne. Collaboration. HackerOne Help Center, 2026 (accessed 2026-11-13). docs.hackerone.com/en/articles/8457618-collaboration (partially unverified)
  • Hardt et al. 2016Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, et al. Strategic Classification. ITCS, 2016. arXiv:1506.06980
  • Holmström & Milgrom 1991Bengt Holmström, Paul Milgrom. Multitask Principal-Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design. Journal of Law, Economics, & Organization 7 (Special Issue): 24–52, 1991. jstor.org/stable/764957
  • Holmström 1982Bengt Holmström. Moral Hazard in Teams. Bell Journal of Economics 13(2): 324–340, 1982. ideas.repec.org/a/rje/bellje/…
  • ICMJE 2026International Committee of Medical Journal Editors (ICMJE). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. ICMJE (updated January 2026), 2026. icmje.org/recommendations
  • Janzing et al. 2020Dominik Janzing, Lenon Minorics, Patrick Blöbaum. Feature Relevance Quantification in Explainable AI: A Causal Problem. AISTATS, PMLR 108: 2907–2916, 2020. arXiv:1910.13413
  • Jiang et al. 2025Z. Jiang, J. Zhang, N. Weir, et al. Core: Robust Factual Precision with Informative Sub-Claim Identification. Findings of ACL: 19833–19856, 2025. doi:10.18653/v1/2025.findings-acl.1018
  • Jiang et al. 2026Pengyi Jiang, Xiaoguang Zhu, Quanyan Zhu. Semantic Cooperative Games for Contribution Attribution in LLM-Based Multi-Agent Systems. arXiv, 2026. arXiv:2607.18255 (preprint)
  • Kumar et al. 2020I. Elizabeth Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, et al. Problems with Shapley-Value-Based Explanations as Feature Importance Measures. ICML, PMLR 119, 2020. arXiv:2002.11097
  • Lee et al. 2025aKiljae Lee, Ziqi Liu, Weijing Tang, et al. Faithful Group Shapley Value. NeurIPS, 2025. arXiv:2505.19013
  • Lee et al. 2025bY. Lee, J. Kim, J. Kim, et al. CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists. EMNLP: 15771–15798, 2025. doi:10.18653/v1/2025.emnlp-main.796
  • Lee et al. 2026Kiljae Lee, Ziqi Liu, Weijing Tang, et al. Priority-Aware Shapley Value. arXiv, 2026. arXiv:2602.09326 (preprint)
  • Li et al. 2021Jiahui Li, Kun Kuang, Baoxiang Wang, et al. Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning. KDD, 2021. doi:10.1145/3447548.3467420
  • Li et al. 2026Yanming Li, Xuelin Zhang, WenJie Lu, et al. Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System. arXiv, 2026. arXiv:2602.08335 (preprint)
  • Lin 2026Z. Lin. Hidden Prompts in Manuscripts Exploit AI-Assisted Peer Review. Communications of the ACM 69(7): 53–56, 2026. doi:10.1145/3779116
  • Linux 2026Linux Kernel Community. AI Coding Assistants. docs.kernel.org (read at 7.4.0-rc2; the Assisted-by tag required in 7.1–7.3 has been removed), 2026 (accessed 2026-11-13). docs.kernel.org/process/coding-assistants.html
  • Liu et al. 2024Zijun Liu, Yanzhe Zhang, Peng Li, et al. A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration. COLM, 2024. arXiv:2310.02170
  • Li, Ren & Yan 2026Ed Li, Junyu Ren, Cat Yan. Scaling Multiagent Systems with Process Rewards. arXiv, 2026. arXiv:2601.23228 (preprint)
  • Lu et al. 2026Mingyu Lu, Yushan Huang, Chris Lin, et al. Agents that Matter: How Subtle Choices Shape Agent Attribution in Multi-Agent Systems. arXiv, 2026. arXiv:2605.27621 (preprint)
  • Ma et al. 2026Ming Ma, Jue Zhang, Fangkai Yang, et al. DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems. ICLR, 2026 (arXiv 2025). arXiv:2512.06749
  • Merton 1968Robert K. Merton. The Matthew Effect in Science. Science 159(3810): 56–63, 1968. doi:10.1126/science.159.3810.56
  • Messick 1995S. Messick. Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist 50(9): 741–749, 1995. doi:10.1037/0003-066X.50.9.741 (partially unverified)
  • Messing 2026S. Messing. Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking. arXiv (author note: accepted to NeurIPS 2026), 2026. arXiv:2604.11581 (preprint)
  • METR 2025Sydney Von Arx, Lawrence Chan, Beth Barnes. Recent Frontier Models Are Reward Hacking. METR Blog, 2025. metr.org/blog/2025-06-05-recent-reward-hacking
  • Mohapatra & Pruthi 2026Arati Mohapatra, Danish Pruthi. Expectations and Practices around AI Disclosure in CS Research. arXiv (accepted to Findings of EMNLP 2026), 2026. arXiv:2608.23271 (preprint)
  • Musco & Witter 2025Christopher Musco, R. Teal Witter. Provably Accurate Shapley Value Estimation via Leverage Score Sampling. ICLR, 2025. arXiv:2410.01917
  • Nature 2023Nature (Editorial). Tools such as ChatGPT threaten transparent science; here are our ground rules for their use. Nature 613: 612, 2023. doi:10.1038/d41586-023-00191-1
  • Nielsen 2008Michael Nielsen. The Future of Science. APS News, November 2008, The Back Page; blog version of the same title. APS News 2008-11 (PDF); Archived blog post
  • NISO 2022NISO. CRediT, Contributor Roles Taxonomy (ANSI/NISO Z39.104-2022). NISO Standard, 2022. credit.niso.org
  • Optimism 2026Optimism. Season 9: From Experiment to Organization. Optimism Blog, 2026. optimism.io/blog/season-9-from-experiment-to-organization
  • Perdomo et al. 2020Juan C. Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, et al. Performative Prediction. ICML, 2020. arXiv:2002.06673
  • Polymath 2009D. H. J. Polymath. A new proof of the density Hales-Jewett theorem. Annals of Mathematics 175(3): 1283–1327, 2012 (arXiv:0910.3926 posted 2009). doi:10.4007/annals.2012.175.3.6
  • Protocol Guild 2026Protocol Guild. Protocol Guild Annual Report 2025. Protocol Guild Blog, 2026. protocolguild.org/blog/20260129-annual-report-2025
  • Rashid et al. 2018Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, et al. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. ICML, PMLR 80: 4295–4304, 2018. arXiv:1803.11485
  • Shapin 1989Steven Shapin. The Invisible Technician. American Scientist 77(6): 554–563, 1989. Harvard DASH scan hdl:1/3425945
  • Shapley 1953Lloyd S. Shapley. A Value for n-Person Games. In H. W. Kuhn, A. W. Tucker (eds.), Contributions to the Theory of Games II, Annals of Mathematics Studies 28: 307–317, Princeton University Press, 1953. doi:10.1515/9781400881970-018
  • Silberzahn et al. 2018R. Silberzahn, et al. Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. Advances in Methods and Practices in Psychological Science 1(3): 337–356, 2018. doi:10.1177/2515245917747646 (partially unverified)
  • Simonsohn et al. 2020U. Simonsohn, J. P. Simmons, L. D. Nelson. Specification curve analysis. Nature Human Behaviour 4(11): 1208–1214, 2020. doi:10.1038/s41562-020-0912-z (partially unverified)
  • SourceCred 2022ezraU. SourceCred (The Organization) Is Winding Down. SourceCred Discourse Forum, 2022. discourse.sourcecred.io/t/…/1383
  • Stenberg 2026Daniel Stenberg. The end of the curl bug-bounty. daniel.haxx.se (personal blog), 2026. daniel.haxx.se/blog/…/the-end-of-the-curl-bug-bounty
  • Stevens 1946S. S. Stevens. On the Theory of Scales of Measurement. Science 103(2684): 677–680, 1946. doi:10.1126/science.103.2684.677
  • Strathern 1997Marilyn Strathern. 'Improving ratings': audit in the British University system. European Review 5(3): 305–321, 1997.
  • Sturmfels et al. 2020Pascal Sturmfels, Scott Lundberg, Su-In Lee. Visualizing the Impact of Feature Attribution Baselines. Distill, 2020. doi:10.23915/distill.00022
  • Sundararajan & Najmi 2020Mukund Sundararajan, Amir Najmi. The Many Shapley Values for Model Explanation. ICML, PMLR 119: 9269–9278, 2020. arXiv:1908.08474
  • Sunehag et al. 2018Peter Sunehag, Guy Lever, Audrunas Gruslys, et al. Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward. AAMAS Extended Abstract (3 pp., from p. 2085), 2018. arXiv:1706.05296
  • SWE-bench 2025Jacob Kahn. Repo State Loopholes During Agentic Evaluation. SWE-bench GitHub issue #465, 2025. github.com/SWE-bench/SWE-bench/issues/465
  • Wang & Jia 2023Jiachen T. Wang, Ruoxi Jia. Data Banzhaf: A Robust Data Valuation Framework for Machine Learning. AISTATS, PMLR 206: 6388–6421, 2023. arXiv:2205.15466
  • Wang et al. 2022Jianhong Wang, Yuan Zhang, Yunjie Gu, et al. SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning. Advances in Neural Information Processing Systems 35, 2022. arXiv:2105.15013
  • Zhang 2026Chenchen Zhang. From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models. arXiv, 2026. arXiv:2604.09459 (preprint)
  • Zhang et al. 2025Shaokun Zhang, Ming Yin, Jieyu Zhang, et al. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. ICML, PMLR 267: 76583–76599, 2025. arXiv:2505.00212
  • Zhuge et al. 2025M. Zhuge, C. Zhao, D. Ashley, et al. Agent-as-a-Judge: Evaluate Agents with Agents. ICML, PMLR 267: 80569–80611, 2025. arXiv:2410.10934

聚沙成洲Grain by grain, new ground.