Experimental results demonstrate that the proposed knowledge-guided hybrid reward framework significantly outperforms existing baselines in both reasoning accuracy and generalization capability, validating the effectiveness and applicability in safety-critical domains.
Abstract
In Reinforcement Learning with Verifiable Rewards (RLVR) frameworks for mathematical reasoning tasks, floating-point results are typically evaluated using a tolerance-based reward. However, this strategy suffers from challenges such as difficulty in threshold calibration, unstable training dynamics, and limited accuracy, especially in clinical scenarios. To address these limitations, we propose a knowledge-guided hybrid reward framework (\textsc{MedCalc-R1}). Specifically, we introduce a knowledge verification reward mechanism that enforces explicit generation of computational formulas, which are further validated by an external verifier to enhance interpretability and reasoning reliability. Furthermore, we design a hybrid soft-hard reward scheme combining a hard constraint based on clinical safety thresholds with a soft, precision-sensitive reward that progressively guides learning within the acceptable range. Experimental results demonstrate that our method significantly outperforms existing baselines in both reasoning accuracy and generalization capability, validating the effectiveness and applicability in safety-critical domains.
Reinforcement Learning with Verifiable Rewards (RLVR) makes Multimodal Large Language Models more accurate, but the gains are brittle: simply paraphrasing a question or changing the prompt template can degrade them, which challenges reliable deployment in high-stakes scenarios like medical VQA. We trace this to two issues of the standard RL objective. First, the binary verifier conflates format with content, so the reward signal cannot tell a wrong answer apart from a misformatted one. Second, the training distribution covers only a thin slice of the real-world prompts that the model might meet at deployment, so policies that perform well on the training distribution can behave differently under unseen prompts during test. Both failures call for a robust post-training method that helps the policy cover a broader distribution of semantically equivalent prompts, and we identify two measures that help achieve this objective: separating format from semantics in the reward, and applying policy invariance across perturbed prompts with equivalent semantics. We therefore propose Prompt-Invariant RLVR (PIRL), consisting of a dynamic trinary reward and a consistency regularizer based on an embedding-space adversary. Under stress testing, PIRL's average accuracy on benchmarks drops by only $\le 1\%$, where GRPO drops ~3%. On dynamic evaluation, PIRL also achieves the smallest performance drop.
P. Zhou, Zhiwei Tang, Xiaopeng Peng et al.· 0 citations
This work introduces StructReward, a compute-efficient framework that provides dense reinforcement signals through structured step-level reward alignment and substantially reduces the computational overhead of multimodal reinforcement learning.
This work reviews pre-2019 XRL approaches, categorizing them into policy explanation, reward decomposition, model transparency, and post-hoc interpretability methods, and proposes a framework that combines interpretable policies, surrogate models, attention mechanisms, and visualization techniques to enhance transparency without significantly reducing performance.
Michael Anderson, David Thompson· International Journal of Art...· 0 citations
When medical AI systems hallucinate clinical reasoning, the consequences extend beyond incorrect answers: fabricated justifications that superficially reference retrieved evidence can mislead clinicians into unsafe treatment decisions. Medical reasoning agents must therefore produce not only correct answers but also faithful justifications that clinicians can verify against cited evidence. We identify a systematic failure mode in RL-trained retrieval agents: outcome-only rewards improve accuracy while degrading faithfulness, a phenomenon we term confident hallucination. The agent learns to answer from parametric memory and backfill plausible but unsupported justifications; citation fabrication rates rise from 16.5% to 31.8% even as accuracy improves by 5 points over the supervised baseline. We address this with a faithfulness-gated reward design: accuracy credit is conditioned on evidence grounding via a hard gate, complemented by retrieval validity and conciseness signals that close exploitation paths unique to agentic retrieval. The resulting system, MedAgent-R1, reduces citation fabrication from 31.8% to 4.7% and raises evidence completeness from 58.7 to 82.6 while maintaining 75.1% accuracy, with 13.2-point gains on HealthBench Safety. Under the same agentic retrieval setup, MedAgent-R1 outscores GPT-4o on faithfulness-specific dimensions (Factual Support 4.55 vs. 4.25; Overclaiming 4.40 vs. 4.15) while remaining below GPT-4o in overall accuracy, suggesting that explicit faithfulness training yields evidence-grounding gains not achieved by scaling alone.
This work proposes neurosymbolic HRL with {\em Incremental Knowledge (InK), where symbolic high-level components perform symbolic planning on an updatable representation of current InK, while low-level goal-conditioned neural modules learn motion primitives through experience using reward shaping.
Subrat Prasad Panda, B. Genest, A. Easwaran· Lecture notes in computer sc...· 0 citations
As large multimodal models (LMMs) become increasingly integrated into clinical workflows, robust evaluation and alignment mechanisms are critical to ensure their reliability in decision support. Existing reward models (RMs) typically reduce preference supervision to scalar scores, limiting transparency and diagnostic insight. In this work, we introduce MedJudge, a multimodal medical reward modeling method that supports interpretable, evidence-grounded, and clinically-aligned decision evaluation. Given a clinical query (text and image) and candidate responses, MedJudge outputs both strict pairwise preferences and concise rationales, enabling transparent verification and inference-time reranking. To overcome the lack of high-quality supervision data, we construct a large-scale synthetic dataset with five complementary generation heuristics, and compare three training paradigms—scalar (BT), supervised (SFT), and reasoning-augmented SFT (SFT-R)—across 8 open-source models. Our comprehensive experiments show that while all methods improve over baseline, SFT-R provides comparable accuracy with richer explanations, and BT achieves the strongest overall performance. Beyond decision accuracy, we propose UMLS-based Concept Overlap (UCO) to evaluate explanation quality, measuring concept-level alignment with clinician expectations. Compared to BERTScore, UCO better reflects human preferences across multiple statistical tests. We further show that our BT-trained judges improve Best-of-4 answer selection by up to 2.75%, demonstrating practical utility in clinical inference-time decision reranking. Our results establish MedJudge as an effective, interpretable, and clinically grounded framework for multimodal medical reward modeling and evaluation. Our code, data and scripts are available at https://github.com/yunhonghe1021/MedJudge.
Yunhong He, Kai Zhang, Jiarong Qian et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
