Do LLMs Explain or Remember? Evaluating Language Model Explanations of Reinforcement Learning Failures

Nivetha Suntharamoorthy Jørn Eirik Betten Dennis Gross Eirik Valseth Helge Spieker

· 1 citation · 26 references

TL;DR

This study systematically investigates how prompt structure, model architecture, and environment representation affect the quality of Explainable RL (XRL) explanations, and conducts a multi-model evaluation using quantitative metrics to assess explanation coherence, helpfulness, and stability.

Semantic Scholar

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Shockingly Simple Self-retrospection Improves Agentic Models Without RL

Retrospection-Only Fine-Tuning (ROFT), a minimal online procedure designed to isolate the effect of explanation-only training on subsequent behavior, is studied, showing that learning to explain can also improve learning to do and motivating further study of explanation-to-action transfer.

Jonathan Light, C. Cui, Jeonghye Kim et al. · 0 citations

#machine learning Preprint Aug 2026

Demystifying Reinforcement Learning Post-Training of Language Models

This work deconstructs the RL post-training algorithm, investigating each step to clarify what is actually happening beneath the surface, and uses the entropy of the policy's output distribution as a lens to compare the distributions learned through pretraining, SFT, and RL post-training, revealing how each stage shape...

D. Clay, Saket Gollapudi, Sankar V Harilal et al. · 1 citation

#machine learning Preprint Aug 2026

Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents

EvalXRL is a benchmark in which a Large Language Model (LLM) coding agent uses different XRL methods to diagnose a held-out malfunction in an RL agent, and then repair it, and proposes the first head-to-head comparison of multiple XRL methods in closed-loop usage.

Ram Rachum, Yotam Amitai, Bálint Gyevnár et al. · 0 citations

#artificial intelligence Review Sep 2026

Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions

This survey provides a systematic taxonomy of recent research on rollout efficiency for reasoning-oriented reinforcement learning, classifying existing approaches from both mechanism and bottleneck perspectives.

N. Gholipour, M. Assunção, Gursimran Singh et al. · 0 citations

#machine learning Preprint Sep 2026

RESCUE: Repairing Language Model Errors to Sparse Circuits via Reinforcement Learning

This work introduces RESCUE (Reasoning-Error Sparse-Circuit Uncovering and Editing), a framework that localizes error-associated circuits and surgically repairs them for performance enhancement, and introduces a pruning technique and precisely fine-tunes error circuits to correct task failures, thereby translating erro...

Chuan-Pu-Zou-Sheng-Li-Guo-Jing-Zhong-Gu-Yue-Shu-Ca Liu, Miao Yu, Yi-Kai Cai et al. · 0 citations

Delayed Convergence and Emergent CoT Reliance in RL-Tuned Language Models

This analysis reveals that intermediate log-probability is an unreliable indicator for reasoning capability; instead, reasoning performance results from a shift of internal confidence allocation where RL fine-tuning delays internal convergence, exhibiting prolonged mid-layer exploration before converging sharply at the...

Pablo Pérez-Lázaro, Rocío Aznar-Gimeno, F. J. Lacueva-Pérez et al. · 0 citations