Task-Oriented RAG Evaluation: Raw Generation and Retrieval Results
This repository contains the raw experimental outputs produced during evaluation of locally deployed large language model (LLM) and retrieval-augmented generation (RAG) systems. The records capture the outputs of the experimental generation pipeline before downstream evaluation metrics were computed. They include standalone chat-mode generations and RAG-mode generations across multiple model, hardware, platform, embedding, and reranking configurations. For RAG runs, each record preserves the retrieved chunks, the subset selected for use after reranking, the generated response, and system-performance measurements. For standalone chat runs, the same experimental logging structure is retained without retrieval-dependent content. The repository also includes the persistent retrieval databases used by the RAG pipeline. These consist of ChromaDB vector stores and Whoosh lexical indexes corresponding to the evaluated embedding and hardware configurations. --- Relationship to the QAC Benchmark The experiments were conducted using a Question-Answer-Citation (QAC) benchmark organized according to ten information-seeking task categories: 1. Reference / Navigational2. Factoid3. Definition / Conceptual4. Summarization5. Procedural / Task-Oriented6. Policy Interpretation / Compliance7. Comparative / Cross-Document8. Analytical / Rationale9. Ambiguous / Exploratory10. Out-of-Scope / Refusal-Required Each experimental observation corresponds to one question variant drawn from the QAC benchmark. Related benchmark dataset: - Task-Oriented RAG Evaluation: Question-Answer-Citation Benchmark Dataset Overview- DOI: 10.5281/zenodo.22677354 --- Repository Organization The raw results are organized by computing platform, hardware execution mode, pipeline mode, generative model, and retrieval configuration. A representative directory structure is: ├── README.md├── evaluation-schema.json└── evaluation_results ├── laptop_1 │ ├── chat │ │ ├── cpu │ │ └── gpu │ ├── db │ │ ├── minilm-cpu │ │ │ ├── chroma │ │ │ └── whoosh │ │ ├── minilm-gpu │ │ │ ├── chroma │ │ │ └── whoosh │ │ ├── mpnet-cpu │ │ │ ├── chroma │ │ │ └── whoosh │ │ └── mpnet-gpu │ │ ├── chroma │ │ └── whoosh │ └── rag │ ├── cpu │ └── gpu └── laptop_2 ├── chat │ ├── cpu │ └── gpu ├── db │ ├── minilm-cpu │ │ ├── chroma │ │ └── whoosh │ ├── minilm-gpu │ │ ├── chroma │ │ └── whoosh │ ├── mpnet-cpu │ │ ├── chroma │ │ └── whoosh │ └── mpnet-gpu │ ├── chroma │ └── whoosh └── rag ├── cpu └── gpu