Adaptive RAG can improve structure compliance and reduce hallucinations in multilingual clinical note generation without dataset-specific prompt engineering or model fine-tuning, and these findings support retrieval-based format adaptation as a generalizable mechanism for diverse clinical documentation contexts.
Abstract
Background/Objectives: Clinical documentation places a significant time burden on healthcare professionals, including in the context of home care. Large language models (LLMs) offer potential for automated note generation, but current approaches rely on static prompt templates that fail to generalize across care settings, languages, and documentation formats. This study proposes and evaluates an adaptive retrieval-augmented generation (RAG) framework that uses retrieval as a format adaptation mechanism, enabling the generation of structured clinical notes from patient–provider transcripts across various documentation formats without model fine-tuning. Methods: The proposed framework retrieves dialogue–note pairs that demonstrate the structure of specific sections, allowing the transfer of formatting knowledge during inference. Experiments were conducted on three datasets covering two languages and various documentation formats: the Japanese Visiting Nurse corpus (JP-VN), MTS-Dialog, and ACI-BENCH. Six controlled conditions were evaluated: zero-shot prompting (C1), static few-shot prompting (C2), dense retrieval (C3), random retrieval (C4), sparse BM25 retrieval (C5), and hybrid retrieval using reciprocal rank fusion (RRF) (C6). Performance metrics include structural adherence to required section headings, content quality (ROUGE-1, BLEU, BERTScore), and the number of hallucinated clinical entities per generated record. Results: Structure compliance increased from 0–37% under static conditions (C1/C2) to 91–100% under all adaptive RAG conditions (C3–C6) across all datasets. On MTS-Dialog, dense retrieval achieved the highest content quality (ROUGE-1: 0.519 vs. 0.446–0.492 for C4–C6; p<0.001). Hallucinated entities in JP-VN decreased from 2.73–3.58 per note (C1/C2) to 1.15–1.30 (C3–C6), an approximately 55–56% reduction. Conclusions: Adaptive RAG can improve structure compliance and reduce hallucinations in multilingual clinical note generation without dataset-specific prompt engineering or model fine-tuning. These findings support retrieval-based format adaptation as a generalizable mechanism for diverse clinical documentation contexts.
Background/Objectives: Large language models (LLMs) show promise for clinical decision support, yet their accuracy in interpreting specialized medical guidelines remains uncertain. Retrieval-augmented generation (RAG) may enhance performance by grounding responses in authoritative knowledge bases. This study aimed to compare the accuracy, comprehensiveness, and safety of RAG-enhanced versus standard LLMs for answering clinical questions derived from the German S3 guideline for oral cavity carcinoma. Methods: We conducted a prospective, single-blind benchmark study evaluating six LLMs: one RAG-enhanced model (Custom GPT with guideline access), one consensus-based model (ConsensusGPT), and four standard models (DeepSeek-V3.2, Mistral Small 3.2, Qwen3-Next-80B, GPT-OSS-120B). Fifty clinical questions covering 17 guideline domains were presented to each model three times, yielding 900 evaluations. Three expert reviewers assessed responses using 5-point Likert scales for accuracy, comprehensiveness, and clarity, under a single-blind procedure, the effectiveness of which was tested by a pre-specified manipulation check. We then ran a paired within-model experiment in which each base model was queried with and without guideline access through a transparent, openly released retrieval pipeline, and scored every response with a condition-blind automated judge alongside deterministic retrieval metrics computed from the logs. Secondary outcomes included hallucination rates and guideline citation behavior. Inter-rater reliability was assessed using intraclass correlation coefficients (ICCs). Results: In a paired within-model design that held each base model fixed, adding transparent guideline retrieval improved accuracy—significantly in the three weaker open-weight models (Mistral, Qwen3, and GPT-OSS) and directionally in the already-strong DeepSeek and GPT-5 bases. Because a pre-specified blinding check found that experts could still identify retrieval-augmented answers with 98.5% accuracy, we anchored causal interpretation on measures that do not depend on the human raters, ranked by their independence: deterministic, log-derived retrieval metrics first, and then an automated, condition-blind LLM judge, whose agreement with the experts (Spearman ρ = 0.81, 95.7% within-one agreement) establishes shared calibration rather than independence from their bias. Deterministically from the retrieval logs, citation groundedness rose from 0% to 51–89% and retrieval recall@5 was 92%. On the judge, content-level hallucination fell from 42% to 4% and accuracy rose by a pooled +0.64 points (95% CI 0.47–0.80); the accuracy gain persisted after adjustment for response length (+0.48, 95% CI 0.22–0.73), which retrieval shortened rather than lengthened. The accuracy gain was large for weaker base models and small or non-significant for already-strong ones, whereas the hallucination and auditability gains were consistent across all models. The human ratings reproduced the judge’s accuracy effect (+0.61, 95% CI 0.49–0.74), and GPT-5 run through the transparent pipeline showed no significant difference from the proprietary Custom GPT (judge accuracy 4.48 vs. 4.58). Conclusions: Guideline retrieval yields a reproducible, largely base-independent improvement in the safety and auditability of LLM answers to clinical guideline questions, with accuracy gains concentrated in weaker base models. Because retrieval-augmented answers are recognizable to experts, rigorous evaluation should rely on rater-independent measures, and residual hallucination continues to require human oversight.
Andreas Vollmer, Lara Schorn, Felix Schrader et al.· Diagnostics· 0 citations
It is suggested that RAG LLMs may be supplementary tools for evidence retrieval and synthesis but cannot, at this time, fully replace comprehensive expert review of the medical literature.
P. A. Krump, Mallory N. Blasingame, T. Koonce et al.· medRxiv· 0 citations
Indonesian healthcare question-answering (QA) systems often operate in low-resource settings and must handle substantial linguistic variability in real-world user queries, including paraphrasing, informal expressions, and implicit intent. These challenges are compounded by limited annotated healthcare data and the diverse ways patients express similar medical needs in everyday language, causing QA models to rely heavily on surface-level wording. This study proposes a new multi-stage data augmentation method to improve the robustness of a healthcare chatbot based on IndoBERT Fine-Tuned. The proposed method integrates domain-specific fine-tuning with an augmentation pipeline that introduces paraphrased question variants with IndoT5, normalizes informal language, and incorporates controlled lexical variation through Part-of-Speech (POS) based verb synonym replacement while preserving medical entities, thereby expanding linguistic coverage without requiring additional manual annotation. The augmentation process preserves medical intent while generating diverse surface forms, enabling the model to learn more flexible representations of user queries. Experimental results demonstrate that the proposed multi-stage augmentation substantially improves the robustness of the IndoBERT-based healthcare question answering system. Full augmentation expands the training data to 1,424 question–answer pairs and achieves 67.37 Exact Match (EM) and 85.59 F1. Compared with training without augmentation, this corresponds to relative improvements of approximately 11.8% in EM and 7.0% in F1, indicating more reliable answer span extraction under paraphrased, informal, and lexically varied queries. These gains reflect improved alignment between conversational user input and structured healthcare information. Overall, this work highlights the importance of integrated data augmentation for enhancing low-resource Indonesian healthcare question-answering systems. By exposing the model to broader linguistic variations during training, the proposed approach supports more stable real-world performance while maintaining medical intent consistency and provides insights into remaining challenges for reliable healthcare QA deployment.
A. Rosyadi, Taufiqur Rohman, Moh. Rizki Fajar et al.· Indonesian Journal of Electr...· 0 citations
Background: Large language models (LLMs) can produce clinically plausible recommendations that are not adequately supported by authoritative evidence. Objectives: We introduce GuidelineGuard, a modular multi-agent retrieval-augmented generation pipeline in which a separate Auditor verifies claim–sentence support before a recommendation is surfaced. Methods: The original evaluation used a 73-sentence guideline corpus and GG-Bench-60, with replication across three open-weight backbones. In response to reviewer concerns about benchmark size and selective evaluation, we added a source-traceable GG-Bench-200 stress test and the complete 500-case held-out PQA-L test split of PubMedQA. The revision experiments compare single-pass RAG, a paired multi-agent no-Auditor ablation, and GuidelineGuard; the paired runner is designed to share the Planner–Retriever–Clinician draft so that the Auditor is the only intervention. Checkpoint verification confirmed an identical observable pre-audit state for all 200 GG-Bench cases and 496/500 PubMedQA cases; four PubMedQA cases were regenerated after quota-interrupted resumption and were correct commitments in both arms. Because the originally used hosted Llama endpoints became unavailable after the initial experiments, the expanded runs use openai/gpt-oss-20b for generation and openai/gpt-oss-120b for the Auditor. Results: On GG-Bench-200, single-pass RAG achieved 0.970 operational accuracy, while the no-Auditor and GuidelineGuard arms achieved 0.955 and 0.925, respectively. GuidelineGuard committed on 186/200 cases (coverage 0.930) and was correct on 185/186 commitments (selective accuracy 0.995); all 186 commitments cited at least one gold evidence identifier. Relative to the paired no-Auditor arm, the gate rejected six otherwise-correct commitments and no incorrect commitment. On PubMedQA-500, single-pass RAG achieved 0.644 operational accuracy at 0.950 coverage, the no-Auditor arm 0.638 at 0.896 coverage, and GuidelineGuard 0.550 at 0.736 coverage. Selective accuracy increased across those operating points from 0.678 to 0.712 to 0.747. Within the 496 PubMedQA cases with verified-identical observable pre-audit state, the gate rejected 36 incorrect and 45 correct pre-audit commitments, demonstrating both error enrichment and a substantial false-rejection cost. Conclusions: The expanded results support GuidelineGuard as a selective claim–evidence verification mechanism, not as a universally more accurate generator. Its value is the explicit, auditable coverage–risk trade-off; the appropriate verification threshold is task- and cost-dependent and requires prospective clinical validation.
