RAGGlass: Beyond the Black Box — A User-Centric Glass Box Framework for Inspecting and Diagnosing Retrieval- Augmented Generation

Abstract

Retrieval-Augmented Generation (RAG) mitigates factual hallucinations in Large Language Model (LLM) workflows by extracting corroborating passages from an external corpus prior to response synthesis. Nevertheless, conventional production pipelines operate as opaque black boxes, obscuring intermediate operations such as text ingestion, sliding-window chunking, vector embedding, similarity indexing, and context curation from end users, which impairs their ability to verify underlying evidentiary support. Although modern observability platforms provide extensive diagnostic instrumentation, they cater almost exclusively to engineering telemetry and developer debugging rather than user-facing inspection. This paper presents RAGGlass, a user-facing framework for improving transparency and interpretability in RAG-based question-answering systems. RAGGlass provides an interactive representation of the RAG workflow through seven observable process states and enables configuration of chunk sizes (100–2,000 characters), chunk overlap (0–500 characters), and Top-K retrieval (1–10). It exposes generated chunks, embedding metadata, 384-dimensional representations from all-MiniLM-L6-v2, and retrieved contexts with similarity and distance scores, including live cosine-similarity comparisons between chunks. ChromaDB supports vector storage and semantic retrieval, while response generation uses the Groq API and Ollama for cloud and local deployment. Evaluated on a 17-question test set (14 answerable, 3 unanswerable probes) across seven chunk sizes (200–800 characters) at Top-K=3, RAGGlass achieved 94.12% accuracy (16/17) for chunk sizes 200, 300, and 500, and 100% (17/17) for sizes 400–800. Faithfulness was 100% across all configurations: all unanswerable probes were correctly declined, and the single accuracy failure was an honest refusal. Chunk-level inspection revealed that smaller chunks split answer sentences across boundaries; increasing Top-K fully recovered performance. RAGGlass thus maintains strong question-answering performance while making retrieval evidence and failure modes directly inspectable to users.```[cite: 3]