# Reading smart vs reading everything

*What it actually costs an agent to answer from your documents – and how we measured it.*

Tokens are the currency of working with AI. Every question an agent answers is paid for in them, and the largest, most avoidable part of that bill is usually not the thinking – it is the *reading*. An agent that swallows an entire folder of documents to answer a question about one of them has paid, in full, for all the pages it didn't need.

The workbench is built to avoid that. Because every document has structure – a table of contents, sections that mean something – an agent can look at the shape of a document first, decide which parts bear on the question, and read only those. We make a plain claim on the strength of it: outline-first reading answers as well as reading everything, for a fraction of the token cost.

This is the evidence for that claim. We ran the test on ourselves.

## Measurements

**Quantity under test.** Retrieval token cost: tokens spent locating and reading source material to answer a question, expressed as a percentage of the tokens required to read the full corpus. Not total inference cost – generation and reasoning tokens, and model choice, are out of scope. This isolates the cost of assembling context, which document structure governs.

**Baseline.** Naive full-fetch within the workbench: same model, same corpus, same questions, each document read in full. No cross-product comparison. The single variable is retrieval strategy.

**Corpus.** Fourteen documents, ~77,000 tokens, from the group's physics programme – one geometric formalism applied from particle physics through to cosmology. Chosen deliberately: the material is novel and absent from any model's training data, so questions are unanswerable without reading the source. Mixed shape by design – two large canonical references, several mid-size notes, a set of small dense filenotes.

**Runs.** Sixteen, three model families, two batches. Batch 1: sixteen lookup questions, each with section-level ground truth. Batch 2: eight cross-cutting synthesis questions requiring assembly across documents. Each question run under two arms – outline-first and full-fetch.

**Controls.** Each run started cold in a fresh thread, no memory of previous runs – Claude incognito; for ChatGPT, whose incognito mode cannot reach the workbench tools, persistent memory disabled. Closed-book arm (no documents supplied) established the floor: near-zero throughout. Costs recomputed deterministically from fetch logs against local corpus copies, not from runner self-report. Clean-rooms verified by recomposition before each spawn; grading against container-verified keys held outside the runner.

## Results

![Two-panel bar chart. Left, simple lookups: full-fetch reads 100% of the corpus, outline-first averages 23% (Claude Fable 22%, Claude Opus 20%, ChatGPT Sol 28%) at identical accuracy. Right, cross-cutting synthesis: full-fetch 100%, outline-first averages 39% (Fable 33%, Opus 35%, Sol 49%) at near-parity quality.](./figures/mdb-34-retrieval-efficiency.svg)

**Lookup.** Outline-first matched full-fetch accuracy exactly – full marks, every model – at roughly a fifth of the token cost. 3.6–4.9× cheaper, no accuracy penalty.

| Simple lookups | Accuracy | Token cost vs full read | Saving | Closed-book |
|---|---|---|---|---|
| Claude Fable 5.0 | 32 / 32 | 22% (≈3.0 of 14 docs) | 4.6× | 0 / 32 |
| Claude Opus 5.0 | 32 / 32 | 20% (≈2.8 docs) | 4.9× | 2 / 32 |
| ChatGPT Sol 5.6 | 32 / 32 | 28% (≈3.9 docs) | 3.6× | 0 / 32 |
| Full read (baseline) | 32 / 32 | 100% (all 14) | 1.0× | – |

*Closed-book: same question, no documents. Near-zero confirms answers came from reading, not training – the two Opus recovered are the general-knowledge exception the corpus was chosen to minimise.*

**Synthesis.** Cost rose with question breadth, as expected: ~2.6× cheaper, quality within 1–2 rubric points of full-fetch. A retrieval strategy that answered a genuinely cross-document question on a fraction of the corpus would be the anomaly. Cost tracks what the question requires.

| Cross-cutting synthesis | Quality (outline-first) | Token cost vs full read | Saving | Quality (full read) |
|---|---|---|---|---|
| Claude Fable 5.0 | 39 / 40 | 33% | 3.0× | 40 / 40 |
| Claude Opus 5.0 | 39 / 40 | 35% | 2.9× | 40 / 40 |
| ChatGPT Sol 5.6 | 38 / 40 | 49% | 2.0× | 40 / 40 |

*Full-fetch edges quality here. Every deducted point traces to one question whose source document had no headings for outline-first to navigate by; where structure existed, the arms tied.*

**Fabrication.** Zero fabricated answers across 528 graded responses, both arms – no claim beyond what the retrieved text supported. On a claim that stands or falls on trust, that figure carries as much weight as the cost.

## Where it fails

The synthesis gap did not spread evenly. It concentrated on a single question whose source document had no internal headings – nothing for outline-first to navigate by, so the relevant passage was unreachable. The method depends on document structure; absent structure, it degrades to blindness, not to a graceful decline.

The edge is therefore sharp and known: outline-first reading is only as good as the structure it reads. Headings make it work; an undifferentiated wall of text defeats it. The limitation sits in corpus authoring, not in the mechanism – which makes it fixable by the author.

## Where full reading still wins

Selective reading is not universally preferable. Full-fetch caught what outline-first could not: the *seams* – the contradiction between two documents, the superseded figure never updated, the term labelled three ways. This is audit work – reconciliation, consistency-checking, detecting what does not line up – and for it, reading everything is warranted.

The result is not that selective reading wins. It is that reading depth should track task demand: lookup costs a lookup, synthesis costs more, audit costs the full corpus. Cost follows requirement.

**These numbers are ours.** They were measured on our physics corpus, under the conditions above – which is the honest limit of what any single evaluation can claim. The interesting question is what the same test says about *your* documents: your filing, your question types, your models.

We will run it. Point us at a representative corpus and we will measure outline-first against full-fetch on your material, to the same protocol, and hand you the numbers – flattering or not.

*Get in touch: [eval@thesis.limited](mailto:eval@thesis.limited).*

*Full method, per-run results, model-by-model breakdown, and the complete findings – including the workbench improvements the evaluation itself surfaced – are in the MDB-34 final report. Sixteen runs, three model families, two days: the workbench measuring itself, and improving from the measurement.*