Single-Round Vector RAG vs an LLM-Compiled Wiki: A Preregistered Comparison on a Small Multi-Domain Research Corpus

Abstract

We preregistered a comparison of two ways to help an LLM answer questions over a small research corpus: single-round Vector RAG and an LLM-compiled markdown wiki browsed by a tool-using agent. Both answered the same 13 questions over 24 papers with the same answer model, scored by two blinded LLM judges. The three preregistered predictions came out one weakly supported, one supported, and one refuted. The wiki scored much better at connecting findings across papers, but its organization advantage fell below the registered threshold once both judges were combined. RAG met the registered test on single-fact groundedness, though the result was judge-sensitive. The wiki was cheaper to build but spent about 21 times more LLM tokens per query, so no break-even point exists. Exploratory analyses bear on why such comparisons disagree. A decomposition-retrieval variant of RAG reduced most of the wiki's synthesis-score gap at lower token cost. The judges' rank agreement was near zero on holistic groundedness (rho = 0.04) against rho = 0.81 on the most concretely defined criterion. A post-hoc claim-level analysis of citation support was checked against two human annotators on 100 claims. Its scorer agreed with them on 50 to 54% of claims as first run and on 65 to 69% once a truncation error was corrected, short of the rule fixed in advance. On the annotators' labels, the analysis did not establish a citation-support advantage for the wiki. On one model and one small corpus, which system appears to win depends on the retrieval baseline, the scoring method, and the judge, so evaluations should report synthesis, citation support, and cost separately and check automated grounding scorers against human labels.