Leveraging Retrieval-Augmented Generation to Support Legal Question Answering: Evidence from Indonesian Human Trafficking Court Decisions

Abstract

The exponential growth of court decisions in Indonesia has posed a crucial challenge for legal practitioners in obtaining relevant information. Conventional search systems fail to capture the in-depth legal context, while Large Language Models (LLMs) are prone to producing hallucinations that can mislead legal reasoning. This study proposes and tests the implementation of Retrieval-Augmented Generation (RAG) to support Legal Question Answering LLM-based Quality Assurance (LQA) to improve factual accuracy. This study used 408 Indonesian court decisions related to criminal cases. Human trafficking data collected and standardized from 143 district courts. The RAG framework is designed in three stages: indexing, search, and incremental generation. We evaluated three Open-source LLM models: Gemma, LLaMA, and Qwen. Three models are also combined with two retrieval methods: BM25 (lexical) and Dense (semantics). Experimental results show that Qwen 3, especially when combined with BM25 RAG, consistently produces the highest overall answer quality across all evaluation metrics (ROUGE and BLEU). The BM25 method is significantly more effective than dense retrieval. Due to the highly standardized nature of court decision documents, Qwen demonstrated peak performance on structured information categories such as “identitas_terdakwa”, achieving a ROUGE-L score of 0.899. The primary contribution of this study is demonstrating that RAG significantly reduces the risk of LLM hallucinations in a legal context. However, error analysis suggests that future improvements should focus on strengthening generation controls to address these issues: unsupported generation and remaining model-rejection behavior.