Semantic Compression Trees is introduced, a hierarchical index in which each node stores only its semantic residual -- the information it adds beyond its parent -- and retrieval proceeds by progressive descent from the root, so that per-query cost is governed by tree depth rather than collection size.
Abstract
Retrieval-augmented generation relies mostly on flat, fixed-granularity indexes: documents are cut into uniform chunks and retrieved by similarity, discarding the hierarchical structure of the source. We introduce Semantic Compression Trees (SCT), a hierarchical index in which each node stores only its semantic residual -- the information it adds beyond its parent -- and retrieval proceeds by progressive descent from the root, so that per-query cost is governed by tree depth rather than collection size. We evaluate on QASPER (50 papers, 173 questions) under two protocols differing only in whether the benchmark supplies the relevant document, with bootstrap confidence intervals and paired significance tests throughout. The results are mixed and we report them as such. When the document is given, SCT with a zero-LLM extractive compressor matches dense retrieval on answer quality (0.274 vs. 0.277 F1, $p = 0.37$) using 30% fewer context tokens and no LLM calls to build the index, and residual storage beats storing full summaries at each node (0.274 vs. 0.205, $p<0.001$). Increasing the collection fifty-fold multiplies flat retrieval's per-query scoring work by 48.9x and SCT's by 6.4x. Progressive descent itself is not supported. Retrieving the same residuals without the tree performs identically when the document is given ($p = 0.27$), and descent is substantially worse when the system must select the document (0.122 vs. 0.165, $p<0.001$). Routing accuracy localises the cause: descent selects the correct paper 20.2% of the time against 39.3% for flat retrieval, because that choice is made from the root residual, the most compressed node in the tree. We conclude that the residual representation is worth keeping and top-down routing is not.
Chunking is the first and most consequential step in retrieval-augmented generation (RAG): every downstream retrieval decision inherits the chunk boundaries. We present a three-stage, chunk-side-only semantic chunking pipeline---header-split, semantic merge, and title-chain prefixing---that costs zero additional LLM calls: the title chain reuses the document's own header hierarchy instead of a generated summary. On a 1600-query stratified evaluation over a production Markdown knowledge base, the pipeline improves MRR@5 from 0.374 to 0.463 (+23.8%) on the full set and from 0.828 to 0.925 (+11.7%) on the answerable subset (n=563), with dual-annotator Cohen's kappa 0.45 (unweighted, 16,000 score pairs). We then report what we tried and what failed: three query-side or architecture-level follow-ups are design dead ends (unevaluated---no comparable run artifacts), one measured failure (prefix weight decay), and one protocol-level failure that is the paper's central methodological finding. In a same-pool prefix on/off ablation, dual-annotator agreement collapsed from kappa 0.45 to 0.04 under identical prompts---stripping the title-chain context strips the disambiguation signal annotators need to agree on relevance. This measurement trap invalidates a common evaluation practice in chunking research and motivates retrieval-time per-candidate prefix evaluation, the direction we recommend from all our evidence.
Retrieval-Augmented Generation (RAG) systems enhance large language models by retrieving relevant documents from external knowledge bases. Recent work by Sarthi et al. (2024) introduced RAPTOR, which organizes documents into hierarchical tree structures for efficient retrieval, but requires expensive LLM-based abstractive summarization at each internal node -- making large-scale deployment prohibitively costly. We present SVD-RAG, the first method to apply Singular Value Decomposition (SVD) on dense sentence embedding matrices for extractive summarization in hierarchical RAG. Unlike classical LSA which operates on sparse TF-IDF matrices, SVD-RAG exploits the rich semantic representations of modern embedding models, identifying the most informative sentences through their energy contribution in the principal components. Our approach is (1) deterministic -- unlike LLM-based summarization, SVD produces identical results for the same input; (2) cost-efficient -- tree construction requires no additional API calls beyond the initial embedding, reducing token consumption by ~85%; and (3) content-adaptive -- the energy-ratio threshold tau automatically adjusts compression based on content complexity. In a controlled head-to-head comparison using identical corpora, clustering, and beam search, SVD-RAG achieves retrieval quality within 1-5% of RAPTOR with LLM summarization (MRR 0.867 vs. 0.875, Recall@1 0.483 vs. 0.458) while building the tree 317x faster (0.1s vs. 31.7s). On a scaled multi-topic benchmark with 205 chunks and 100 queries across 20 topic variations, SVD-RAG achieves a 4.2x improvement in Recall@1 and 3.1x improvement in MRR over flat embedding retrieval. We provide a detailed cost analysis and parameter sensitivity study. Our implementation is released as an open-source Python package.
This work introduces AnchorQE, a training-free method that separately encodes the original query and its expansion before interpolating them, and shows that AnchorQE improves retrieval effectiveness by up to 12.89% when compared to widely-used expansion-only or text-level concatenation baselines across TREC-DL, LoTTE, and BEIR.
Retrieval-Augmented Generation (RAG) has improved the factual grounding of large language models; however, conventional retrieval strategies remain limited for long-document question answering, as relevant information is often distributed across multiple document sections and may be inferential rather than lexically similar to the query. This paper proposes a reasoning-aware hierarchical traversal mechanism that interleaves chain-of-thought generation at each depth-first search node evaluation step, incorporating the resulting reasoning embedding into a combined node scoring function alongside query similarity. The hierarchical tree is constructed through iterative chunking, embedding, clustering, and summarization; retrieval is then guided by this combined score under a dual-threshold pruning mechanism that adaptively controls traversal depth and breadth. The proposed approach is evaluated on the NarrativeQA and QuALITY benchmarks against a semantic similarity-based traversal baseline. On QuALITY, the method achieves marginal gains in overall accuracy (+0.7%) and F1 (+0.1%), while substantially reducing the abstain rate from 4.1% to 1.8%, with improvement concentrated on normal-difficulty questions; on hard questions, the baseline outperforms the proposed method. On NarrativeQA, ROUGE-L F1, BLEU-1, and BLEU-4 improve slightly, though METEOR decreases, reflecting inconsistent metric-level effects. These results suggest that step-wise CoT reasoning can improve node selection quality in hierarchical traversal, with gains most evident on standard-difficulty questions. Performance on hard questions and metric-level consistency across NarrativeQA remain open challenges, indicating that the quality of the generated reasoning signal is a key bottleneck for further improvement.
Experimental results show that SAC-RAG reduces token consumption by 38%–58% at the cost of only a 1–2 percentage point EM drop, with EM actually improving after compression for reasoning-type questions, achieving the optimal quality–efficiency trade-off in terms of token consumption.
Deyu Zhang, Hongqiang Yu, Jinze Huo et al.· IEEE Access· 0 citations
A policy-aligned retrieval framework that improves offline relevance over a matched-capacity baseline, with gains broadly distributed across facet combinations, and serves this framework with a two-stage GPU architecture.
Dhritiman Das, Chujie Zheng, Ronak Kaoshik et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
