Sep 2026· Companion Publication of the 28th International Conference on Multimodal Interaction· 0 citations· 30 references
Computer Science
Abstract
Many of the qualities that matter most in how people learn and grow, how someone regulates their emotions, reflects on a setback, or stays aware of others during a difficult conversation, are not directly observable. They have to be inferred from how someone speaks, moves, and sounds over time, and they resist the kind of clean labeling that most machine learning pipelines are built around. We study this challenge through a case that is well grounded in psychological theory but rarely modeled computationally: self-compassion, the tendency to respond to one’s own setbacks with patience rather than harsh self-criticism. We examine how it appears during structured reflective interviews in a technology-mediated training setting, where people naturally talk through socio-emotionally demanding situations. Since no existing dataset captures this kind of construct in this kind of setting, we collected and annotated 51 reflective dialog sessions using an independent, temporally overlapping annotation scheme grounded in established theory. We consolidate the underlying six-component psychological model into a three-class supervision space, balancing self-kindness and mindfulness against self-critical or overwhelmed states, and build a reproducible window-based pipeline that aligns video, audio, and text on a shared timeline. Unimodal models trained on each modality separately are compared against a simple probability-level fusion strategy, which yields modest but consistent gains over the best single modality. We close by discussing where each modality succeeds or struggles, what this suggests about how this kind of construct is actually expressed in reflective speech, and what would be needed to model it, and constructs like it, more effectively.
RelationalDialogues is introduced, a novel, fully synthetic dataset of 12,849 multi-turn dialogues designed to explicitly train perspective-taking, demonstrating that training on highly contextualized, metadata-driven synthetic data is an effective method for advancing LLMs from displaying superficial sympathy to engag...
Neema Owji, Cameron Buckner· UF Journal of Undergraduate...· 0 citations
Large language models are increasingly consulted at moments of distress, yet single-turn benchmarks neither test sustained exchanges nor distinguish between users. We built a personality-aware evaluation in which four widely used models advised several synthetic help-seekers, each given a psychometrically specified pro...
Pablo Fonseca, R. Rodríguez-Carvajal, Rafael A. Calvo· 0 citations
It is found that PST is shaped by identity development and demographic factors, and the proposed Human-AI Self-Talk (HAIST) framework is proposed to guide the design of non-clinical AI companions to support learning and self-sufficiency.
Ann-Kareen Gedeus, N. Wagener, Ying-Xuan Li et al.· Proceedings of the 14th Nord...· 0 citations
Affective computing has progressed from categorical emotion recognition to open-ended affective analysis with large multimodal models. Yet affective science describes emotion as an unfolding process shaped by appraisal, regulation, and social interpretation, which remains underexplored computationally. We propose TRACE...
SPIN is introduced, a cognitive-affective personality system-inspired inference pipeline for simulating human social-psychological behavior that suggests that structured personality-state inference can improve benchmark-level behavioral alignment beyond richer persona descriptions or generic multi-step reasoning.
Zhi-Bo Deng, Dong-Yuan Li, Shu-Wen Ge et al.· 0 citations
Detecting true felt emotions when speakers suppress or mask their internal state poses a fundamental challenge for affective computing systems. We study a specific form of affective dissonance in dyadic speech: utterances where a speaker’s self-reported emotion diverges from all external observer ratings, indicating th...
Cheng-Shiuan Lin, T. Eze, Da-Wei Xie et al.· Proceedings of the 28th Inte...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 28, 2026
Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026