#machine learning #natural language processing #human-computer interaction Preprint

ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents

Xiaoyu Wang Qing-Qing Gu Yue Zhao Teng Chen Yuqi Cao Xiaokai Chen Hong-Yan Li Luo Ji

Aug 2026 · 0 citations · 59 references

Computer Science

TL;DR

A two-level hierarchical reinforcement learning (HRL) framework for conversational agents that bridges the gap between existing token-level and utterance-level RL methods and introduces a dual-granularity reward mechanism that combines the utterance-level satisfaction score with token-level intrinsic self-consistency and a KL-divergence penalty.

Abstract

Humans naturally exhibit multiple forms of abstraction in reasoning and interaction, including temporal abstraction across decision timescales and strategic abstraction over communicative intents. Inspired by these complementary abstractions, we propose a two-level hierarchical reinforcement learning (HRL) framework for conversational agents that bridges the gap between existing token-level and utterance-level RL methods. Built upon a two-level Markov decision process (MDP), our framework conditions token-level response generation on utterance-level actions represented by explicit textual strategies. Based on theoretical analysis and efficiency considerations, we employ DQN to optimize the high-level Q-network and PPO to train the low-level actor-critic. To further alleviate reward sparsity and facilitate convergence, we introduce a dual-granularity reward mechanism that combines the utterance-level satisfaction score with token-level intrinsic self-consistency and a KL-divergence penalty. Experiments on both daily-life and emotional support conversations demonstrate that our method consistently outperforms a wide range of baselines in both strategy determination and response quality. Our implementation is available at https://github.com/AaronJi/ToSCA.

DOI arXiv Semantic Scholar

View source

Similar papers

#artificial intelligence Preprint Sep 2026

OSPD: On-Policy Self-Distillation for Persona-Consistent Dialogue

This work proposes OSPD, an on-policy self-distillation framework where the same model serves as both teacher and student under asymmetric information: the teacher receives a complete character profile while the student sees only a brief summary, and the student generates trajectories from its own policy.

Rui Xu, Yikai Zhang, Ai-Li Chen et al. · 0 citations

#machine learning Preprint Aug 2026

Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL

This work extends contrastive reinforcement learning (CRL), a prototypical self-supervised method, to operate over action chunks, and finds that this results in large, pervasive gains across established offline and online benchmarks: +31.7% and +93.1% across 18 and 11 environments respectively.

Michal Korniak, Kamil Dybek, Benjamin Eysenbach et al. · 3 citations · ⚡1

#natural language process... Preprint Sep 2026

SocialRL: Refining LLMs'Social Intelligence through Multi-turn Reinforcement Learning and Reward Design

This work proposes SocialRL, a multi-turn reinforcement learning framework using PPO that propagates delayed outcome rewards back to each turn, enabling long-horizon planning and demonstrates the effectiveness of SocialRL across synthetic and real social scenes, as well as standard and challenging social scenarios.

Jia-Ning Wang, Xin-Tao Wang, Ai-Li Chen et al. · 1 citation

#artificial intelligence Preprint Sep 2026

StateTree: Enhancing Long-Term Dialogue Reasoning via Reinforcement Learning

Large language models deployed as personalized assistants must reason over long, evolving interaction histories. However, in long-term dialogue reasoning, relevant evidence is scattered across sessions, preferences may be revised over time, and standard long-context training fails to address these challenges under data...

Na-En Xu, Wan-Qing Cui, Yi-Bo Hu et al. · 0 citations

#machine learning Preprint Sep 2026

Dyad: Extending Large Language Models with Native Typed Decision-Making

Dyad is introduced, an architecture that augments a pretrained LLM with an environment-conditioned action encoder that embeds each candidate action description in parallel, then scores these embeddings against the LLM's internal state to yield a distribution over typed actions.

Yundaichuan Zhan, Wei-Shi Wang, Wen-Biao Liu et al. · 0 citations

#machine learning Preprint Oct 2026

AURAL: Adaptive Latent Reasoning with Joint Chunk for Speech Language Models

Model intelligence and fast response jointly shape the quality of interaction with speech language models, yet remain difficult to achieve together. Explicit chain-of-thought (CoT) improves reasoning and audio understanding, but generating intermediate reasoning tokens delays responses. Describing fine-grained acoustic...

Yu-Xiang Wang, Kun-Yu Feng, Yuan-Cheng Wang et al. · 2 citations