Does Document Structure Still Pay?

Not long ago that approach was simply impossible. A flagship context window held a fraction of this 282,039-token novel: the question fit, the book did not. So people worked around it, and put only the relevant parts into the window instead of the whole document.

Then the windows grew: to hundreds of thousands of tokens, and now to a million and more, some roughly two million. The whole novel sits inside. The founding reason was gone, but the workarounds did not disappear.

The workaround family got a name.

R, retrieval: find the few pieces of the document that matter for this question. A, augment: put those pieces next to the question, inside the model's context window — this is the prompt the model will read. G, generate: the model answers from what it was given, and only from that.

There are dozens of published RAG variants, and each one cuts, condenses or connects the document differently. Following the grouping in the Gao et al. survey, they recur as three families: cut into chunks, condense into a summary tree, or connect into an entity graph.

The study benchmarks a widely used representative of each family.

Back to the naive approach: the whole novel and the question, composed in one context window. In this study it is the baseline, and it has a name. Flat: full-context prompting, no structuring at all.

The worry is that every question re-reads the same unchanged text, and every re-read bills the full price. Then providers started caching. The first question still pays the full read, and that read fills the cache; from the second question on, re-read text is billed at a quarter of the fresh price.

One worry stays. The answer is a single line somewhere in that wall of text, and models can miss details buried in very long prompts.

Back to the two axes from the start: whatever the company runs is judged on quality and on cost. Once the document fits, and re-reading is cheap, does structure still pay?

Three research questions make it precise. Which approach gives the best quality for the cost. Which structural features explain the differences. And does the one-time build pay off. That open question is, word for word, the title of the thesis, and everything from here on is the answer.

Two datasets. QASPER: 249 research papers carrying 955 questions, scored with Answer-F1. NovelQA: 55 full novels carrying 1,381 questions, scored as multiple-choice accuracy.

They differ where it matters: length. Papers run a few thousand tokens; novels reach ~770,000, and the average novel is 46 times the length of the average paper.

I brought the library's copy of Robinson Crusoe to the defense. On the benchmark side it is one NovelQA item: 282,039 tokens, answered by all four architectures. The answer is a single sentence, somewhere in those tokens.

Before the four pipelines, one frame to read them all with. Every method splits into the same two phases. Build time reads the document once and turns it into a reusable artifact: chunks, a summary tree, an entity graph, or, for the baseline, nothing at all. Query time consults that artifact instead of the raw novel, and the model answers from what it returns.

The split is also the shape of the bill: the build is paid once per document, the querying on every single question.

All four architectures face that one Crusoe question. The meters below are the NovelQA averages, in thousandths of a dollar (m$): what each pays once to build a novel's artifact, and what it pays on every question after that.

Flat and Naive RAG answer correctly; RAPTOR and GraphRAG agree on the same wrong option. Build once, then answer: the same meters, side by side, in thousandths of a dollar per novel and per question.

One anecdote proves nothing. The study measures the pattern: 2,336 questions, 5 repeats each.

Everything an architecture spends fits on one ruler: what it pays once per document to build, plus the small charge on every single question, forever. Input the model has already read bills at a quarter of the fresh price, and that discount is priced into every number here.

Now remember the setting. The document is fixed and the questions keep coming, so the build is a pie that every new question cuts into more slices. The slice only shrinks, it never reaches zero, and chunk and tree retrieval stay cheaper per question too; GraphRAG's per-question pipeline already costs more than Flat's cached re-read of a short paper. Whether a build ever fully pays for itself depends on how many questions a document really gets, and that is measured, not guessed.

One shared answerer, gemini-3.1-flash-lite-preview, at temperature 0. One scorer. Every question answered 5 times.

That is the whole setup. Behind every number that follows: 2,336 questions, each one answered 5 times by every architecture.

Each dataset's native metric, Answer-F1 on the papers and accuracy on the novels, with 95% document-clustered confidence intervals. The dashed line is the model with no document at all. On the novels that floor is high, because famous novels are partly memorized, but the ranking survives it: reading the book lifts Flat by +0.16 over the floor and GraphRAG by only +0.01. On the papers the floor sits at 0.16, far below every score.

Three claims. The order is the same on both datasets: Flat, then Naive RAG, then RAPTOR, then GraphRAG, with one statistical tie in the middle on each (Naive RAG and RAPTOR on the novels, RAPTOR and GraphRAG on the papers). Flat scores highest everywhere, significantly so except for its narrow lead over Naive RAG on the papers, which survives the confidence interval but not the multiple-comparison correction. And adding structure never improves quality.

Each point is one architecture's quality against everything it spends on the whole dataset. The efficient frontier is two points: Flat for quality, Naive RAG for budget. The tree and the graph are dominated, because another architecture is better on one axis and no worse on the other.

Same picture on the novels, at wider margins. Dominated on both datasets.

The whole Crusoe question hangs on one sentence of the novel: “I observed the tears run down his face”. Whoever keeps that wording can answer; whoever loses it cannot. So ask every architecture the same thing: does that sentence reach the window?

A build is an up-front investment, repaid only if enough questions arrive per document. Papers ask about 4 questions each, and RAPTOR's build is repaid only past 9, so on the papers the investment never pays; GraphRAG never gets below Flat's line at all.

Novels ask about 25 questions each, and there the picture changes. RAPTOR's investment is repaid at about 16, inside what the dataset actually asks, but only on the standard price card: a deeper cache discount pushes the break-even to 28, above it. GraphRAG would need about 71 questions per novel before its build pays back.

Four honest threats. One: another answerer might rank the four differently, since a single model family carried the study, and the pilot's rank-stability gate is a mitigation, not proof. Two: only papers and novels were tested, so beyond those two datasets the verdict is extrapolation.

Three: each architecture ran one fixed configuration, so the comparison is the claim, not the absolute levels. Four: the model partly knows famous novels by heart, and the closed-book control bounds that; the ranking is unaffected.

Inside that scope, the verdict holds. And every threat doubles as the next experiment.

Best quality for the cost: the frontier is two points, Flat for quality and Naive RAG for budget. What explains the differences: whether the answer's wording survives into what the model reads. Does the build pay off: only RAPTOR, only on novels, only on the standard price card, and even a repaid build buys no quality.

One sentence carries the study. With long-context models, added document structure does not pay, for fixed documents answering repeated questions.