KD-GAG: Knowledge-distilled graph-augmented generation via preference-aware subgraph pruning.
TL;DR
A novel framework KD-GAG is proposed that efficiently constructs KGs by distilling teacher LLMs' semantics and reasoning into a smaller student LLM and a preference-based subgraph pruning method to optimize the retrieval process.
Abstract
Retrieval-Augmented Generation (RAG) methods that integrate external knowledge sources have demonstrated significant effectiveness in addressing the knowledge scarcity and hallucination issues of Large Language Models (LLMs). Compared to traditional approaches relying on documents as knowledge sources, Knowledge Graphs (KGs) enable GraphRAG to emerge as a superior solution through their capability to capture complex inter-dependencies. However, existing GraphRAG methods still exhibit limitations: insufficient attention to optimizing KG construction, introduction of excessive irrelevant information during recall optimization in retriever, and inadequate consideration of the generator's data preferences during retrieval refinement. In this paper, we propose a novel framework KD-GAG that efficiently constructs KGs by distilling teacher LLMs' semantics and reasoning into a smaller student LLM. Furthermore, we propose a preference-based subgraph pruning method to optimize the retrieval process. This method automatically learns subgraph pruning strategies through answer quality feedback and aligns the retriever's selection mechanism with the generator's knowledge preferences. Extensive experiments on four QA benchmarks, including HotpotQA, 2WikiMQA, NQ, and MedHop, demonstrate that KD-GAG achieves strong and consistent performance across multi-hop QA, open-domain QA, and biomedical reasoning settings.