Aug 2026· Cognitive Computation· Vol 18· 0 citations· 64 references
TL;DR
This work discovers persona-centric metamorphic relations to infer test samples from annotated data, without additional annotation cost, and evaluates the robustness of personalized dialogue models regarding persona consistency, revealing that prompt learning is more robust than training from scratch and fine-tuning.
Abstract
Retrieval-based dialogue systems aim to select a proper response according to multi-turn conversational history. Persona-based conversation utilizes prior knowledge to maintain persona consistency, enhancing retrieval accuracy. However, reference-based evaluation relies on high-quality data annotation, which is costly and time-consuming. To address this, we discover persona-centric metamorphic relations to infer test samples from annotated data, without additional annotation cost. Benefiting from this, this work efficiently evaluates the robustness of personalized dialogue models regarding persona consistency. Specifically, we discover three types of metamorphic relations from three aspects: self-persona, partner-persona, and response, to automatically derive new test samples . Then the inherent inference relations between originals and derivatives allow for robustness evaluation. Using this evaluation methodology, our work assesses three widely used training paradigms: non-pretraining, fine-tuning after pre-training, and prompt learning, in personalized dialogue retrieval to observe whether these paradigms are more robust or exhibit the same flaws as the other two paradigms. Our experimental results, based on the three discovered metamorphic relations with consistent outputs reveal that prompt learning is more robust than training from scratch and fine-tuning. While traditional reference-based validation and natural language processing methods achieve competitively high retrieval accuracy (Hits@1 up to 87.4%), the persona consistency of dialogue retrieval systems is just 20.98% when persona descriptions are perturbed using various metamorphic relation-based transformations.
This work analyzes real chatbot failures to identify six recurring mechanisms and defines six controlled evaluation modes in Hy-MultiTurn, a Chinese benchmark for deep multi-turn dialogue understanding, which shows that Hy-MultiTurn is broadly challenging.
Eileen Ye, Ji-Hua Tao, Yao-Ming Li et al.· arXiv.org· 0 citations
An unsupervised fine-tuning pipeline that harvests reasoning trajectories via in-context learning inference via in-context learning inference is proposed, enabling Large Language Models (LLMs) to access external knowledge and produce factual responses.
The RecSys Challenge 2026 studies conversational music recommendation as a joint item recommendation and response generation problem: given a multi-turn dialogue, systems must retrieve relevant tracks from a large catalog and produce a grounded natural-language response. This paper presents the challenge task, dataset, evaluation protocol, and official results. Beyond the leaderboard, we analyze the 16 accepted systems through a common retrieve–rerank–generate framework and examine how recommendation performance varies across users, requests, and dialogue contexts. Strong systems commonly combine heterogeneous candidate sources and preserve source-specific evidence for learned reranking. Across the system papers and our organizer-side analysis, robust design also means 1) grounding cold-start retrieval in multi-turn conversation and item signals, 2) using intent detectors, and 3) modeling the full multi-turn context rather than the current query alone. We further identify limitations of the benchmark and evaluation protocol, including single-ground-truth relevance and teacher-forced evaluation of synthetic dialogues. Together, these findings provide practical guidance for future conversational recommender systems and shared evaluation efforts.
Seungheon Doh, Sergio Oramas, B. Sguerra et al.· Proceedings of the Workshop...· 0 citations
Despite recent advances in dialogue topic segmentation, existing work provides limited evidence about why individual utterances are predicted as boundaries and how local explanations depend on the selection strategy and perturbation protocol. Using fixed checkpoints of 3LHSeg, a hierarchical dialogue topic segmentation model, we evaluate a boundary-centered framework that compares five local utterance-selection strategies through complementary perturbation diagnostics. The audit also examines configuration sensitivity and explanation stability and uses Random-Baseline Gain (RBG) as a diagnostic of relative local distinctiveness by contextualizing comprehensiveness against matched random subsets from the same local candidate window. Experiments on TIAGE, QMSum, and Friends show that Leave-One-Out ranks most favorably under the adopted perturbation-based diagnostics, although this result is protocol-specific. The diagnostics provide partially overlapping information and vary with the local configuration. Raw deletion area-under-the-curve values are consistently positively associated with initial boundary confidence, but a centered control substantially attenuates this association in most dataset–strategy combinations. Overall, explanation behavior, probability quality, confidence, stability, and boundary correctness should be treated as distinct dimensions, supporting multi-diagnostic and configuration-explicit auditing rather than reliance on a single explanation score.
Fayçal Nouar, H. Belhadef· Electronics· 0 citations
This paper presents a novel scalable framework for multi-domain dialogue state tracking that leverages the pretrained BERT model to achieve zero-shot generalization, making it easy to quickly adapt to new domains without additional training.
Junyuan Zheng, O. Salvi, John Chan· arXiv.org· 0 citations
In persona-based dialogue generation (PDG), LLMs often overuse persona attributes by incorporating them regardless of dialogue context, resulting in unnatural responses. Despite its practical significance, the underlying causes remain unexplored, with no method to mitigate this problem or metric to assess the appropriateness of persona use. To address these issues, we first conduct a comprehensive analysis of LLM-based PDG, revealing that LLMs exhibit a systematic bias to incorporate all given persona attributes, and that existing metrics fail to capture contextual appropriateness. Building on these findings, we propose Self-CONtrastive Persona Overuse Suppression (SCONPOS) to mitigate overuse by directly intervening in LLMs'internal representations at the prompt encoding stage, without requiring any response generation. We further propose the Persona Appropriateness Score (PAS), a novel metric that penalizes both overuse and underuse. Experimental results demonstrate that SCONPOS systematically reduces overuse, and PAS captures the contextual appropriateness of persona use.
