Personalizing LLMs for Financial Regulation Using RAG and Knowledge Graphs: A Case Study at Regnology

Abstract

Financial regulatory documents are characterized by their fine-grained complexity and pronounced heterogeneity, featuring specialized domain-specific content and diverse structural formats that vary across different regulatory frameworks and jurisdictions. These characteristics challenge modern Question-Answering (QA) systems, which often suffer from limited domain adaptation, poor interpretability, and hallucinatory problems. This work was conducted within a banking and software company, where such challenges directly impact regulatory compliance efforts. Our goal is to introduce RegulQA, a hybrid QA system that can extract accurate, logical, and comprehensible answers from unstructured regulatory documents. RegulQA integrates knowledge graph reasoning, semantic search, and retrieval-augmented generation using large language models. Experimental evaluation shows that RegulQA improves QA performance and significantly reduces hallucination rates. The baseline model employing only the LLM demonstrated a hallucination rate of 24%, whereas the proposed approach, combining knowledge graph reasoning and semantic retrieval with LLM reasoning, effectively reduced the hallucination rate to approximately the half. This integrated approach also yielded the best balanced overall scores across key qualitative attributes, including coverage, non-redundancy, readability, and response quality.