Building per-query payouts for third-party RAG data

All of the code is on GitHub.

The proposed transaction

A conventional retrieval-augmented generation (RAG) application searches a corpus its developer has already assembled or licensed. datagraph implements the mechanics for a different model: retrieve relevant records from independent data providers for one question, then pay those providers from the query fee.

This market doesn’t exist yet. The repository runs the transaction locally with synthetic providers, SQLite, and internal credits. It doesn’t implement provider discovery, pricing, access control, or an external payment rail. It does implement the part that has to work before any of those systems can settle a query: deciding how much of one payment each provider earned.

The difficult case is overlapping data. In the demo, borealis and cascade supply the same fact. Remove either provider and the answer doesn’t change because the other still supplies it. A leave-one-out score therefore pays both providers 0, even though the answer depends on their shared fact. Its weights total 0.2744 rather than 1.0000, and normalizing the result transfers the missing share to the providers with unique data.

datagraph uses Shapley attribution by default. It measures each provider across different provider orderings, so redundant providers share credit, and the weights add up to the full payment before they are converted to credits.

What we built

A local provider registry. Providers register datasets with field-level disclosure policies. Retrieval searches only disclosed content. OPEN fields pass through, DERIVED fields are coarsened, and HIDDEN fields are removed before a prompt or query result is built.

A query transaction. One payment enters escrow before retrieval or model work. Records from at least 3 provider accounts are required. Every exit path either settles the escrow or refunds it.

An attribution loop. The model answers from all retrieved providers, then answers again from provider subsets. Each subset is scored against the full answer. Providers, not records, are the units being measured.

A ledger. Contribution weights become integer credits through largest-remainder apportionment. Every ledger entry balances, and settlement is rejected unless its payouts exhaust the escrow.

The offline path is one command:

git clone https://github.com/ashnkumar/datagraph && cd datagraph
uv run datagraph compare

compare runs the same question under sampled Shapley, exact Shapley, and leave-one-out. No API key, service, or container is required. datagraph demo prints retrieval, redaction, the generated answer, the payouts, and the final ledger state. Adding --live uses the Anthropic API.

Lesson 1: A contribution score isn’t necessarily a payment split

We started with leave-one-out because it asks a direct question: how much worse is the answer without this provider?

def leave_one_out(players: Sequence[str], value: ValueFn) -> Attribution:
    grand = frozenset(players)
    total = value(grand)
    weights = {p: total - value(grand - {p}) for p in players}
    return Attribution(engine="leave_one_out", weights=weights, grand_value=total)

v(S) measures how much of the full answer can be produced from provider subset S. The reference answer from all providers is pinned to v(N) = 1, so a leave-one-out weight is the value lost when one provider is removed.

The resulting weights don’t have to add up to 1. Redundancy makes them fall short. Complementarity makes them exceed it.

In the demo, borealis and cascade disclose identical records. Removing either one changes nothing, so both weights are 0:

Those payout rows all total 1000 credits because the allocator normalizes its input. The weight row shows what happened before normalization. Leave-one-out accounted for only 0.2744 of the answer, then the allocator expanded the positive weights to fill the escrow. aurora and delta received the share that the redundant pair lost.

The opposite failure is just as possible. If an answer requires 3 complementary providers, removing any one can collapse the answer. Each provider then receives a leave-one-out weight of 1.0, for a total of 3.0. Normalization divides every measured share by 3. The same arithmetic that inflates a shortfall dilutes an excess.

Shapley attribution avoids both cases by measuring each provider in different arrival orders. A provider supplying a redundant fact receives credit when it arrives first and none when it arrives after the duplicate. A complementary provider receives only its marginal contribution in each ordering. Every ordering starts with no providers and ends with all providers, so its marginal contributions add up to the full value. Their average does too.

The default engine samples 2000 orderings. exact_shapley computes the same allocation by enumerating the subset space. Sampling changes how credit moves between providers, but each sampled ordering still adds up to the whole. That is why the sampled demo pays the redundant pair 195 and 203, while exact Shapley pays 197 each, and both weight totals remain 1.0000.

A balanced ledger doesn’t validate the weights

Shapley efficiency applies to the raw weights. A separate edge case appears when one provider has a negative marginal contribution because adding its records makes an intermediate answer worse.

A negative contribution can’t become a debt, so the first implementation floored it to 0. The remaining positive weights then summed to more than the payment. The allocator normalized them, the escrow emptied, and every ledger invariant passed. The accounting was correct after the attribution had already been changed.

The current query path checks before allocation:

        if result.clamped_excess > EFFICIENCY_TOLERANCE:
            return self._refund(
                query_id,
                question,
                escrow,
                researcher,
                sources,
                f"{self.engine} scored at least one provider below zero; the shares that "
                f"remain claim more than the payment, and settling would mean scaling them "
                f"to fit",
                attribution=result,
                answer=answer,
                model_calls=value.calls,
            )

The query is refunded rather than silently reducing the positive shares. Settlement can prove that money wasn’t created, lost, or stranded. It can’t prove that the attribution rule assigned the right weights before settlement began.

Lesson 2: Providers are the unit; records aren’t

The first player definition was one player per retrieved record. That is easy to implement and easy to manipulate. A provider can divide one contribution into more rows and receive more total credit.

We reconstructed that design on the seeded fixture. Cloning one of delta’s 2 records 4 times moved its payout from 446 to 612 credits of 1000, a 37% increase without new information.

The shipped implementation makes each provider one player. A provider enters or leaves a subset with all of its retrieved records:

    def _generate(self, coalition: frozenset[str]) -> str:
        self._calls += 1
        subset = [s for s in self.sources if s.provider_id in coalition]
        return self.model.answer(self.question, subset)

Retrieval also suppresses exact duplicates from the same provider and limits how many result slots one provider can occupy:

        records = self.registry.search(
            question,
            limit=self.max_sources,
            max_per_provider=max(1, self.max_sources // self.cohort_floor),
        )

With 6 result slots and a cohort floor of 3 providers, one provider can occupy at most 2 slots. Adding 10 exact copies of every delta record leaves its exact-Shapley payout on 365 credits.

This protects the row boundary, not the identity boundary. Splitting delta’s 2 records across 2 provider accounts changes their combined payout from 365 to 446. The attribution layer sees 2 players because the registry supplied 2 identities. Preventing that requires verified identity or a cost to registration below the scoring system.

The general rule is straightforward: when the paid party controls how many scoring units exist, the unit definition is part of the security model.

Lesson 3: An exact total can still contain a noisy split

Attribution regenerates answers under different source conditions. Ideally, the sources would be the only variable. A live language model doesn’t provide that guarantee.

While building the live path, a request with non-default temperature, top_p, or top_k returned a 400 from Claude Opus 5 — thinking is enabled on every request here, and those parameters can’t be set alongside it. There is no seed parameter. More importantly, deterministic decoding was never a guarantee: two calls with identical inputs can still return different text.

Turning thinking off created a different measurement risk. With thinking disabled, internal tags can appear in visible output. Here, visible output is the value being scored, so an internal tag isn’t a cosmetic defect; it changes provider payouts. The shipped request keeps adaptive thinking enabled at low effort and omits thinking text from the response:

        request: dict[str, Any] = {
            "model": self._model,
            "max_tokens": self._max_tokens,
            "thinking": {"type": "adaptive", "display": "omitted"},
            "output_config": {"effort": self._effort},
            "system": SYSTEM_PROMPT,
            "messages": [{"role": "user", "content": build_prompt(question, sources)}],
        }

Two implementation choices keep the arithmetic stable:

1. Every provider subset is generated once and cached for the duration of the query.
2. The grand coalition reuses the reference answer, which pins v(N) to exactly 1 instead of comparing the reference with a second sample of itself.

Neither makes the split deterministic. A provider’s marginal contribution is still the difference between independently generated answers. Wording variation can be credited to whichever provider was added in that comparison. The weights still add up to the payment, but which provider receives them can move between runs.

The offline model exists to separate those claims. It deterministically returns the union of facts in the supplied records, so the test suite exercises the real attribution code with known outcomes. The live path is a contribution estimate from one generation per provider subset; it doesn’t compute an uncertainty interval.

There is another way the generator can change: server-side refusal fallbacks can return an answer from a different model. Mixing models inside one attribution run changes 2 variables at once. The response includes the model that served it, so AnthropicModel compares that value with the requested model. A mismatch ends the query with a refund.

Exact settlement and accurate attribution are separate claims. This project guarantees the first for accepted Shapley settlements. A live model makes the second an estimate.

Lesson 4: Provider count controls model-call cost

The default sampled engine walks 2000 provider orderings, but it doesn’t make 2000 * n model calls. v(S) depends on the set of providers in S, not the order that reached it, so scores are cached by provider subset.

The demo has 4 providers. There are 2^4 = 16 possible subsets, including the no-records baseline. The sampler reaches the same 16 cached subsets as exact Shapley, so sampling costs the same number of generations while producing a slightly noisy split.

Leave-one-out uses n + 2 generations: one removal per provider, the full answer, and the no-records baseline. On the demo, that’s 6 calls against 16.

The number of providers is therefore the main cost control. Retrieval returns at most 6 records, so a default query can contain at most 6 providers and 64 subsets. Raising that cap makes exact enumeration grow exponentially. Sampling saves calls only when it’s stopped before the visited subsets saturate the full space.

One of those generations is intentionally empty. The no-records answer establishes a similarity floor. Generated answers share boilerplate even when they share no facts; without subtracting that floor, stock phrasing would be counted as provider contribution. The extra generation is what makes a provider that changes nothing receive a weight of 0.

Prompt caching doesn’t reduce the demo’s input cost. The source list changes near the beginning of every user message, and the measured requests are shorter than the model’s minimum cacheable prefix. On a larger workload, stable material should precede the varying records so provider subsets can share a prefix. On this fixture, subset memoization is the cache that matters.

What this doesn’t solve

• Provider accounts aren’t verified identities. Multiple accounts controlled by one operator are treated as multiple players.

• The scorer measures lexical overlap. TokenF1 can detect whether content survived, but it doesn’t establish semantic correctness or answer quality.

• Privacy is application-enforced. Hidden fields don’t enter the prompt or QueryResult, but the raw records remain in SQLite and the live path sends disclosed values to the model provider.

The registry, credits, and providers are local. A production marketplace would still need provider connectivity, authentication, pricing, durable settlement, and an external payment rail.

Try it

git clone https://github.com/ashnkumar/datagraph && cd datagraph
uv run datagraph compare

Source: https://github.com/ashnkumar/datagraph