Aug 2026· Journal of Biomedical Informatics· pp.
105095
· 0 citations· 18 references
Medicine
TL;DR
For institution-constrained clinical QA, in-domain teacher-supervised fine-tuning was the most reliable evaluated adaptation path, and public-benchmark transfer was not a dependable proxy for local utility in the primary internal comparison.
Abstract
Objective
To evaluate whether small language models adapted on public medical benchmarks transfer reliably to institution-constrained open-answer clinical QA, and to identify which adaptation choices-supervision format, optimization objective, and backbone-determine local answer quality and evidence coverage.
Methods
We used DistilGPT2 (82M) as the primary student and Llama3 70B as the teacher. We compared four strategies: public-benchmark answer-only adaptation, public-benchmark chain-of-thought adaptation, in-domain teacher-supervised question-answer fine-tuning (QAFT), and direct preference optimization (DPO). We evaluated each strategy on cleaned public benchmarks and on an internal EHR-grounded open-answer task using token-F1, exact match, hallucinated evidence rate, and evidence coverage. We tested robustness via multi-seed reruns, a controlled DPO pair-construction ablation, a hard-negative DPO variant, and cross-backbone replication on TinyLlama 1.1B and a modern Qwen2.5-3B model.
Results
Public-benchmark adaptation did not produce stable gains in repeated-seed external evaluation. In the primary internal comparison, in-domain QAFT outperformed public-benchmark transfer (F1 0.1310 vs 0.1090). DPO did not improve answer quality uniformly; instead, it shifted models toward shorter, stricter responses with lower evidence coverage. In a controlled fixed-split three-run ablation, this shift varied with rejected-response construction. TinyLlama replication showed backbone-specific DPO effects.
Conclusion
For institution-constrained clinical QA, in-domain teacher-supervised fine-tuning was the most reliable evaluated adaptation path. Public-benchmark transfer was not a dependable proxy for local utility in the primary internal comparison, and DPO should be interpreted as operating-point control with local validation before deployment. We propose a local evaluation protocol: adapt on the target task, validate DPO as an operating-point control, and interpret groundedness jointly with evidence coverage.
Developing artificial intelligence capable of clinical language comprehension and reliable diagnostic reasoning has remained a core challenge in biomedical engineering. While Large Language Models (LLMs) demonstrate significant potential in general natural language processing tasks, their direct application in the medical domain is severely constrained by parametric hallucinations and data silos. This paper introduces an end-to-end, resource-efficient, multilingual speech-driven Question-Answering (QA) framework optimized for localized clinical support. To accommodate deployment on consumer-grade execution environments, we implement Parameter-Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA) and 4-bit Quantized LoRA (QLoRA) configurations across open-source 3B and 7B parameter architectures. Human preference alignment is enforced via a stateful Reinforcement Learning with Human Feedback (RLHF) loop applying Proximal Policy Optimization (PPO). Crucially, to mitigate the vulnerabilities of passive information retrieval, we introduce an Active Validation Loop powered by Corrective Retrieval-Augmented Generation (CRAG). This validation engine is decoupled from the model harness using the Model Context Protocol (MCP), standardizing asynchronous lookups across dense vector repositories, clinical guidelines, and real-time electronic health registries.
Misha Patel· International Journal of Sci...· 0 citations
Background/Objectives: Large language models (LLMs) show promise for clinical decision support, yet their accuracy in interpreting specialized medical guidelines remains uncertain. Retrieval-augmented generation (RAG) may enhance performance by grounding responses in authoritative knowledge bases. This study aimed to compare the accuracy, comprehensiveness, and safety of RAG-enhanced versus standard LLMs for answering clinical questions derived from the German S3 guideline for oral cavity carcinoma. Methods: We conducted a prospective, single-blind benchmark study evaluating six LLMs: one RAG-enhanced model (Custom GPT with guideline access), one consensus-based model (ConsensusGPT), and four standard models (DeepSeek-V3.2, Mistral Small 3.2, Qwen3-Next-80B, GPT-OSS-120B). Fifty clinical questions covering 17 guideline domains were presented to each model three times, yielding 900 evaluations. Three expert reviewers assessed responses using 5-point Likert scales for accuracy, comprehensiveness, and clarity, under a single-blind procedure, the effectiveness of which was tested by a pre-specified manipulation check. We then ran a paired within-model experiment in which each base model was queried with and without guideline access through a transparent, openly released retrieval pipeline, and scored every response with a condition-blind automated judge alongside deterministic retrieval metrics computed from the logs. Secondary outcomes included hallucination rates and guideline citation behavior. Inter-rater reliability was assessed using intraclass correlation coefficients (ICCs). Results: In a paired within-model design that held each base model fixed, adding transparent guideline retrieval improved accuracy—significantly in the three weaker open-weight models (Mistral, Qwen3, and GPT-OSS) and directionally in the already-strong DeepSeek and GPT-5 bases. Because a pre-specified blinding check found that experts could still identify retrieval-augmented answers with 98.5% accuracy, we anchored causal interpretation on measures that do not depend on the human raters, ranked by their independence: deterministic, log-derived retrieval metrics first, and then an automated, condition-blind LLM judge, whose agreement with the experts (Spearman ρ = 0.81, 95.7% within-one agreement) establishes shared calibration rather than independence from their bias. Deterministically from the retrieval logs, citation groundedness rose from 0% to 51–89% and retrieval recall@5 was 92%. On the judge, content-level hallucination fell from 42% to 4% and accuracy rose by a pooled +0.64 points (95% CI 0.47–0.80); the accuracy gain persisted after adjustment for response length (+0.48, 95% CI 0.22–0.73), which retrieval shortened rather than lengthened. The accuracy gain was large for weaker base models and small or non-significant for already-strong ones, whereas the hallucination and auditability gains were consistent across all models. The human ratings reproduced the judge’s accuracy effect (+0.61, 95% CI 0.49–0.74), and GPT-5 run through the transparent pipeline showed no significant difference from the proprietary Custom GPT (judge accuracy 4.48 vs. 4.58). Conclusions: Guideline retrieval yields a reproducible, largely base-independent improvement in the safety and auditability of LLM answers to clinical guideline questions, with accuracy gains concentrated in weaker base models. Because retrieval-augmented answers are recognizable to experts, rigorous evaluation should rely on rater-independent measures, and residual hallucination continues to require human oversight.
Andreas Vollmer, Lara Schorn, Felix Schrader et al.· Diagnostics· 0 citations
Across medical benchmarks, MedLLM shows a pattern visible only at sub-billion scale: medical competence does not degrade uniformly under compression but splits by task type and dissociation is masked at 7B, where both capabilities are present, and surfaces only when capacity is scarce.
M. R. Rahman, Asim Ahmed, Mihan Mohagheghzadeh et al.· arXiv.org· 0 citations
An engineering-oriented, end-to-end roadmap that structures the full lifecycle of clinical language model systems—from model design and domain adaptation to optimization and real-world evaluation is introduced.
Overall, the proposed reformulation module produces more natural and clinically relevant Arabic questions and can be plugged into Arabic clinical QA pipelines as a measurable, tunable front-end.
Walid Ounachad, M. Khenchouch, Imad Zeroual et al.· Language Resources and Evalu...· 0 citations
Results indicate that a single serialization-based paradigm, without bespoke fusion architectures, is sufficient for multimodal clinical prediction - substantially reducing system complexity while matching or exceeding specialized designs.
A. Ravichandran, Bilgin Osmandoja, Klemens Budde et al.· arXiv.org· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduSep 14, 2026
The “HardFlow” algorithm could help generative AI models produce high-quality outputs that obey strict requirements when “pretty close” doesn’t cut it.
AI may appear weightless, but every model depends on physical infrastructure. To understand responsible AI, we need to look beyond algorithms and consider the entire lifecycle of the hardware behind them. The post Responsible AI Must Consider Its Afterlife appeared first on GPT-Lab.
