Science Engine · experiments ledger

Does splitting the reading across agents save tokens?

Every test here was written down before it ran: the question, the arms, the decision rule and the predictions, then frozen, run once, audited and reported whatever the answer. This page is generated from those receipts.

Pilot 010 · persistent specialists

On a corpus larger than one agent's window, do eight persistent specialists (each holding one file as cached state, addressed by a code catalogue) process fewer tokens over forty questions than the best single retrieving agent, at the same quality, with their loading charged, and does the saving vanish when the same specialists are made stateless?

Processed tokens over the question stream

Cumulative tokens the model actually evaluated (prompt tokens after cache reuse, plus generated), charged per arm. The persistent arm starts at the cost of loading its eight files; the question is where its line crosses under the generalist's and stays there.

The three arms

G · retrieving generalist

One slot. For every question the code index picks the one or two files that hold the quoted names, and the agent reads contract + files + question. It keeps the caching every single agent gets: its contract prefix is reused on every call.

S · persistent specialists

Eight slots, one file each, loaded once. A question goes to the file's own clerk, who reads only the question. Cross-file questions use a hand-off: a record request to the second file, then the question with that record as a fact.

S0 · stateless specialists

The same protocol as S with caching off, so every call re-reads its file. This is what every earlier pilot measured; it isolates persistence as the cause of any saving.

Every experiment so far

Try it with your own shape of data

An estimate, not a run: it applies the per-token costs measured in pilot 010 to the shape you describe, using the same arithmetic the arms follow (the generalist re-reads the files it retrieves; a specialist reads its file once and then only the questions). Paste your own files to use their real sizes. A real run on your data needs the GPU box; that queue is the next thing to build.

Or paste your own files

One file per block, separated by a blank line. First line is the file name; each following line is name, value. Only the sizes are used; nothing leaves your browser.

How to read the labels

SUPPORTED
The preregistered decision rule passed, including its isolation check. Says exactly what the rule says and nothing more.
NOT SUPPORTED
The rule failed on the thing it tested: quality, or the cost measure.
UNINFORMATIVE
The run completed cleanly but the numbers fell between the pass and fail thresholds, or the saving did not isolate to the named cause.
NOT EVALUABLE
Something about the run itself (accounting, attestation, an interrupted journal) means the numbers cannot carry the claim either way. Reported, never rerun quietly.

Receipts live in the repository: a preregistration per experiment, a frozen copy with the identity of the model, server binary and data, a journal of every call, an analysis receipt and an independent read-only audit.