AffectSim: A Controllable Interactive 3D Simulation Benchmark for Embodied Affective Perception
TL;DR
The results show that affective perception depends not only on the observed behavior, but also on how evidence is acquired, and by making observation acquisition experimentally controllable, AffectSim provides a foundation for studying embodied affective perception in interactive 3D environments.
Abstract
Existing affective benchmarks largely rely on fixed recordings, where observation conditions are predetermined before inference. Consequently, they mainly evaluate passive affect recognition while overlooking a key question for embodied agents: how to actively acquire informative affective evidence. We introduce AffectSim, a controllable interactive 3D simulation benchmark for embodied affective perception. AffectSim represents affective behaviors as replayable 3D episodes and separates the underlying behavior from how it is observed, enabling systematic control over distance, orientation, occlusion, scene geometry, and agent viewpoint. It contains 27,647 episodes across five emotion categories and 57 scenes. Across 24 frozen perception-model configurations, we show that observation quality substantially affects recognition. To examine whether active sensing can recover this gap, we establish a training-free heuristic baseline, which improves recognition in 21 of 24 settings. Building on this, we propose Evidence-Aware Observation Gate (EAOG) to adaptively assess whether additional observation is still beneficial. These results show that affective perception depends not only on the observed behavior, but also on how evidence is acquired. By making observation acquisition experimentally controllable, AffectSim provides a foundation for studying embodied affective perception in interactive 3D environments. All benchmark assets, code, and simulation environments will be released.
