Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· pp. 11005-11015· 0 citations· 9 references
Abstract
Large language models (LLMs) possess extensive latent knowledge yet remain largely static at inference. Once prompted, their generation policy typically cannot evolve, and post-hoc ''self-reflection'' methods provide no explicit principled learning signals. To address this limitation, we formally model iterative research idea optimization as a finite-horizon Markov Decision Process and propose InfRL (Inference-time Reinforcement Learning), a framework designed for effective policy improvement at inference without updating model weights. InfRL coordinates three specialized LLM agents: (i) a State Transition Agent that proposes candidate ideas; (ii) a Policy Update Agent that learns feedback strategies based on idea trajectories; and (iii) a Reward Agent that assigns normalized, comparative rewards, enabling nuanced reinforcement signals. This empowers the Policy Update Agent to dynamically refine feedback strategies, progressively leveraging latent knowledge encoded within the LLM. We evaluate InfRL on five balanced datasets covering 500 recent research papers from health, genetics, environment, neuroscience, and engineering domains. Compared to a single-pass GPT-4o baseline and a strong self-reflection baseline, InfRL improves the novelty of generated ideas by 3.93%--32.7% and the feasibility of generated ideas by 70.1%--448.6% with GPT-4o, exhibiting consistent improvements across ten inference-time iterations. Ablation studies underscore the critical roles of our reward formulation and modular agent design, while human evaluations confirm alignment between reward trends and perceived idea quality. InfRL thus offers a practical and domain-agnostic approach to harness reinforcement learning during inference, bridging the gap between static prompting and computationally intensive parameter-level fine-tuning. The code and the dataset we use are provided at: https://github.com/amir-hassan25/InfRL
This work proposes TAPO: Transition-Aware Policy Optimization for LLM Agents, a unified training framework that alternates between policy optimization and transition supervision, and demonstrates that TAPO consistently improves task performance over pure policy optimization baselines.
Objectives: To develop a framework that integrates pedagogical structure with uncertainty-aware decision-making for personalized learning in smart educational environments, addressing the limitations of current deep reinforcement learning approaches that treat curricula as unstructured sequences. Method: The study formalizes the learning domain as a concept lattice—an order-theoretic structure derived from formal concept analysis that encodes prerequisite relationships. Within this structured state space, a Bayesian reinforcement learning agent using Thompson sampling maintains joint posterior distributions over the learner's latent knowledge state and the uncertain reward associated with each instructional action. The framework was evaluated on the ASSISTments 2012-2013 dataset (4,317 problems, 112 knowledge components, 334,416 interactions) and Eedi (98 concepts, 7,547 interactions)—and validated against four baseline methods: Standard Thompson Sampling, Graph-Constrained RL, Bayesian RL, and Static Policy. Findings: The proposed Structured Thompson Sampling (STS) framework achieved a 15.2% improvement in average skill gain over standard Thompson sampling on ASSISTments and a 14.8% improvement on Eedi, demonstrating consistent performance across datasets. The system demonstrated faster convergence with approximately 32% fewer training interactions. The system outputs well-calibrated uncertainty estimates with an expected calibration error of 0.036, supporting interpretable decision-making for educators. The Pedagogical Coherence Score of 0.96 confirms that STS respects prerequisite relationships, while ablation studies revealed that both the lattice structure and Bayesian optimization contribute significantly to performance. Novelty: This work presents the first integration of formal concept analysis with Bayesian reinforcement learning for pedagogical sequencing, providing a mathematically rigorous foundation for personalized learning that combines structural validity with quantifiable confidence estimates. The framework bridges the critical gap between pedagogical coherence and uncertainty-aware decision-making in adaptive educational systems.
Keywords: Bayesian Reinforcement Learning, Personalized Learning, Concept Lattice, Thompson Sampling, Pedagogical Sequencing, Uncertainty Quantification, Smart Learning Environments, Adaptive Educational Systems
S. Ahamed, A. R. Mohamed Shanavas· Indian Journal of Science an...· 0 citations
Behavior prior reinforcement learning (BPRL) has emerged as a promising paradigm to improve sample efficiency in online reinforcement learning (RL) by leveraging policy priors derived from offline demonstrations. However, most existing BPRL methods rely on static offline datasets, which often suffer from low data diversity and suboptimal trajectory quality. This reliance restricts the effectiveness of policy priors, hindering both policy exploitation and stability during online training. Consequently, agents are prone to inefficient exploration and unstable learning dynamics. To address these limitations, we deviate from existing offline pretraining methods and propose an expert behavior prior (EBP) algorithm. In particular, we introduce a Q-guided conditional variational autoencoder (Q-CVAE) that learns to generate expert policy priors directly from the online replay buffer. This enables the generation of high-value actions for guiding policy updates without relying on precollected expert trajectories. To further enhance policy exploitation, we propose an expert policy guidance (EPG) mechanism that selects expert actions from a generative support set, and we integrate a policy gradient correction (PGC) module to harmonize Q-guidance with expert supervision, promoting stable and consistent policy improvement. Extensive experiments conducted on robotic control (Gym, PyBullet) and industrial control (DMControl) benchmarks demonstrate that EBP significantly outperforms state-of-the-art online RL algorithms, achieving higher sample efficiency and more stable convergence.
G. Gao, Weidong Zhao, Xianhui Liu et al.· IEEE Transactions on Neural...· 0 citations
SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model, is proposed.
Jinyang Wu, Shuo Yang, Zhengxi Lu et al.· arXiv.org· 7 citations
This work proposes AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning that aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space.
Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao et al.· 1 citation
This work introduces SPOT (Sampling Policy Observation Tree), a novel model-agnostic, sampling-based framework for interpreting DRL policies, and demonstrates how its tree-based representation can be used to inspect policy preferences, compare alternative future trajectories, and reveal downstream behaviors that are not visible through single-timestep feature-attribution methods.
Tamar Gozlan, Claudia V. Goldman· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
