Skip to content

Multimodal Detection of Higher-Order Behavioral Constructs: Self-Compassion in Structured Reflective Interactions

Sep 2026 · Companion Publication of the 28th International Conference on Multimodal Interaction · 0 citations · 30 references
Computer Science

Abstract

Many of the qualities that matter most in how people learn and grow, how someone regulates their emotions, reflects on a setback, or stays aware of others during a difficult conversation, are not directly observable. They have to be inferred from how someone speaks, moves, and sounds over time, and they resist the kind of clean labeling that most machine learning pipelines are built around. We study this challenge through a case that is well grounded in psychological theory but rarely modeled computationally: self-compassion, the tendency to respond to one’s own setbacks with patience rather than harsh self-criticism. We examine how it appears during structured reflective interviews in a technology-mediated training setting, where people naturally talk through socio-emotionally demanding situations. Since no existing dataset captures this kind of construct in this kind of setting, we collected and annotated 51 reflective dialog sessions using an independent, temporally overlapping annotation scheme grounded in established theory. We consolidate the underlying six-component psychological model into a three-class supervision space, balancing self-kindness and mindfulness against self-critical or overwhelmed states, and build a reproducible window-based pipeline that aligns video, audio, and text on a shared timeline. Unimodal models trained on each modality separately are compared against a simple probability-level fusion strategy, which yields modest but consistent gains over the best single modality. We close by discussing where each modality succeeds or struggles, what this suggests about how this kind of construct is actually expressed in reflective speech, and what would be needed to model it, and constructs like it, more effectively.

Read PDF

Similar papers

Open access Sep 2026

Beyond Labels: Training Cognitive Empathy with Context-Rich Synthetic Dialogues

RelationalDialogues is introduced, a novel, fully synthetic dataset of 12,849 multi-turn dialogues designed to explicitly train perspective-taking, demonstrating that training on highly contextualized, metadata-driven synthetic data is an effective method for advancing LLMs from displaying superficial sympathy to engag...

Neema Owji, Cameron Buckner · 0 citations
Preprint Aug 2026

All four leading LLMs talk more than they listen to personality-verified synthetic help-seekers

Large language models are increasingly consulted at moments of distress, yet single-turn benchmarks neither test sustained exchanges nor distinguish between users. We built a personality-aware evaluation in which four widely used models advised several synthetic help-seekers, each given a psychometrically specified pro...

Pablo Fonseca, R. Rodríguez-Carvajal, Rafael A. Calvo · 0 citations
Book Open access Oct 2026

Talk to Me Kindly: Designing AI Companions to Foster Positive Self-Talk in Young Adults

It is found that PST is shaped by identity development and demographic factors, and the proposed Human-AI Self-Talk (HAIST) framework is proposed to guide the design of non-clinical AI companions to support learning and self-sufficiency.

Ann-Kareen Gedeus, N. Wagener, Ying-Xuan Li et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Cognition-Oriented Emotion Tracing from Causes to Consequences in Real-World Social Scenes

Affective computing has progressed from categorical emotion recognition to open-ended affective analysis with large multimodal models. Yet affective science describes emotion as an unfolding process shaped by appraisal, regulation, and social interpretation, which remains underexplored computationally. We propose TRACE...

Hao Li, Jin-Ye Zhang, Bobo Li et al. · 0 citations
Preprint Oct 2026

Enhancing LLMs with Cognitive-Affective Personality Inference for Simulating Human Social-Psychological Behavior

SPIN is introduced, a cognitive-affective personality system-inspired inference pipeline for simulating human social-psychological behavior that suggests that structured personality-state inference can improve benchmark-level behavioral alignment beyond richer persona descriptions or generic multi-step reasoning.

Zhi-Bo Deng, Dong-Yuan Li, Shu-Wen Ge et al. · 0 citations
Book Open access Oct 2026

Unmasking Suppressed Emotion: A Multi-Agent Approach to Affective Dissonance

Detecting true felt emotions when speakers suppress or mask their internal state poses a fundamental challenge for affective computing systems. We study a specific form of affective dissonance in dyadic speech: utterances where a speaker’s self-reported emotion diverges from all external observer ratings, indicating th...

Cheng-Shiuan Lin, T. Eze, Da-Wei Xie et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Oct 28, 2026

Making machines that can listen

Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.