MDKeyChunker: What Does One LLM Call per Chunk Buy for Markdown Retrieval?

Abstract

Markdown carries structure a parser reads for free: headers, section paths, and block boundaries. Many RAG pipelines also spend LLM calls per chunk on generated metadata. We ask what one LLM call per chunk buys over that free structure. MDKeyChunker splits Markdown into header-led chunks without splitting any block; makes one LLM call per chunk for a title, summary, keywords, entities, questions, and a subtopic key, showing the model the keys already assigned in the document (a rolling key dictionary); and can merge same-key chunks. With qwen2.5:7b, we evaluate 79 Qasper questions over 30 papers and 73 FreshStack questions over 24 Laravel documentation files under BM25, two dense embedders, and hybrid fusion, following an analysis plan committed before results were computed. Evidence is matched only against source text, within a fixed token budget. Under hybrid retrieval, structural chunks beat 512-character windows on both datasets (Qasper +23.0 points, 95% CI [+12.8, +33.5]; FreshStack +5.1 [+1.4, +9.0]) and 256-token windows on Qasper (+12.7 [+5.3, +20.3]) but not on FreshStack (-2.6 [-6.4, +1.2]). Under the primary retrievers (hybrid, BM25), enrichment shows no planned-comparison difference from a free section-path prefix or from contextual retrieval; under hybrid retrieval the intervals exclude gains above 4.5 and 2.3 points on Qasper and 6.1 on FreshStack. Outside the planned comparisons, enrichment-style prefixes help BM25 on Qasper (exploratory) and mxbai on FreshStack (a secondary retriever). Rolling keys raise key reuse from 5.5% to 14.7%, but merging does not improve retrieval, and under BM25 on Qasper merging with rolling keys scores below merging without them (-6.0 [-13.1, -0.2]). Enrichment used about 1,000 input tokens per chunk; contextual retrieval 5,520 (Qasper) and 9,825 (FreshStack). The results of versions 1 and 2 are withdrawn.