The results suggest that broad SFT brings most of the model's capability improvement; turn-local supervision can be effective when failure detection is precise, with observed transfer concentrated primarily within-family.
Abstract
Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, and choose valid actions under changing constraints. We study this setting in the LM Playschool Challenge with a 2B open-weight model, and find that many failures are not only broad knowledge failures but also local decision failures: repeated guesses, malformed actions, and violations of feedback that the model has just seen. These diagnostics motivate a training recipe organized around three steps: acquire broad game participation through supervised fine-tuning, repair mechanically verifiable failures within one targeted dialogue-game family using turn-local preference pairs, and preserve general capabilities beyond these dialogue games. In the official final evaluation, our submission improves public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, while approximately preserving aggregate static performance (44.14 vs. 44.24 for the baseline). Out-of-domain clemscore remains low at 7.88, with the largest gains concentrated in unseen variants of the targeted family. Our results suggest that broad SFT brings most of the model's capability improvement; turn-local supervision can be effective when failure detection is precise, with observed transfer concentrated primarily within-family.
It is suggested that imitating full trajectories helps with playability, while turn-level and teacher-guided training usually improve decision-making and increase the overall score, and small models are performant simply by using careful curation strategies rather than aggressive changes.
AnthroDial is presented, a closed-loop framework that formulates anthropomorphic dialogue as a joint problem of system architecture, executable evaluation, and diagnostic alignment and shows that anthropomorphic dialogue benefits when generation, evaluation, and reward shaping share the same behavioral dimensions.
Wentao Liu, Si-Yu Song, Xi Chen et al.· arXiv.org· 0 citations
This work analyzes real chatbot failures to identify six recurring mechanisms and defines six controlled evaluation modes in Hy-MultiTurn, a Chinese benchmark for deep multi-turn dialogue understanding, which shows that Hy-MultiTurn is broadly challenging.
Eileen Ye, Ji-Hua Tao, Yao-Ming Li et al.· arXiv.org· 0 citations
Reinforcement learning from human feedback (RLHF) has shown strong potential for aligning language models, but its role in task‐oriented dialogue (TOD) remains unclear. In TOD, models are typically trained with local turn‐level supervision, while system behavior is evaluated through broader interaction‐level properties. This mismatch becomes more challenging in online settings, where explicit dialogue‐level rewards and human preference annotations are unavailable. In this work, we study whether RLHF can be usefully applied to TOD under this limitation. We consider two task‐annotation regimes, partially annotated and fully annotated TOD data, and construct pseudo‐preference pairs using empirical ranking heuristics motivated by prior work on synthetic feedback and model‐based ranking signals. We then train reward models on the constructed pairs and optimize dialogue policies with Preference policy optimization (PPO) using simulator‐generated online trajectories. Experiments on MultiWOZ 2.1 show that the proposed RLHF approach consistently improves corpus‐based evaluation over supervised baselines, while simulator‐based effects remain mixed and backbone‐dependent.
Textual skills provide a lightweight way to improve frozen language-model agents, but their self-evolution normally requires a stable validation signal. Such signals are natural in mathematics or code, where an answer can be checked after it changes, yet are problematic in open-ended dialogue: changing the assistant response also changes the user's next reaction, so a logged reaction cannot directly evaluate a counterfactual response. We propose future-feedback skill evolution, which first redirects self-evolution from prescribing the current answer to predicting whether the observed answer will lead to a positive or negative subsequent user signal. This prediction task is verifiable on fixed logged tuples and therefore supports validation-gated textual optimization. The evolved feedback skill captures interpretable criteria for response quality and can subsequently serve as a diagnostic and optimization target for answer skills. On a proprietary, privacy-preserving sales-assistant dataset, careful quality filtering and a balanced resolved/unresolved split yield more than 75% prediction accuracy. Beyond this result, the central contribution is a formulation that converts otherwise moving conversational feedback into a fixed offline learning target, enabling reproducible skill evolution without placing every candidate skill in live traffic. We discuss the boundary between observational verification and counterfactual validity, and position the method as an offline optimization stage rather than a replacement for final human or online evaluation.
Large language models can interpret user requests plausibly yet fail to produce actions that satisfy structural constraints. We study this problem in Dungeons & Dragons (D&D) combat, where player intent must be grounded in the current game state and translated into well-formed commands for Avrae, a Discord-based D&D automation system. We fine-tune a 4-bit quantized LLaMA-3 8B model with Low-Rank Adaptation on 256 curated FIREBALL-derived examples and compare it with zero-shot, one-shot, and few-shot prompting. On 25 held-out interactions, the model achieves 56% semantic grounding accuracy and 48% strict command validity. Results suggest that parameter-efficient adaptation improves grounding, while formatting, argument, and reference errors remain common.
Ez Zwall, Kedong Chen, Yalun Zhou et al.· Proceedings of the 26th ACM...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
