A Methodology to Compare Real and LLM-Simulated Multimodal Interactions: The Case of Gender Bias

Abstract

Large Language Models (LLMs) are increasingly used to generate multimodal data for building corpora or simulating synthetic behaviours. A key challenge remains the evaluation of generated data. Comparison with real multimodal human interactions may help address this issue, but unified representations and evaluation frameworks are still lacking. In this article, we introduce MAMBI (Multimodal Annotation and Measurements for Bias in Interactions), an open-source tool designed to facilitate systematic comparisons between real and LLM-simulated multimodal human interactions. For this purpose, a standardised enriched multimodal transcript format is proposed, providing a common representation of verbal, prosodic, and non-verbal behaviours. This representation is obtained through an automatic annotation pipeline that transforms raw audio-visual recordings into structured narrative transcripts by combining speech transcript, segmentation, prosodic and facial feature extraction. The pipeline is applied on multiple multimodal corpora, demonstrating its portability across diverse interaction settings and different numbers of participants. Based on this enriched multimodal transcript, a framework is introduced to assess the statistical consistency between real and generated data, focusing on distributional similarity and correlation structure preservation in the prosodic, interactional dynamics, and facial expressions features, with a particular attention on the bias analysis to quantify group-related disparities. As a use case, we present in this article an analysis of gender bias on a large dyadic interaction corpus. The results show that enforcing distributional constraints substantially improves alignment with real data, whereas structural consistency depends more unevenly on the prompting strategy and is not systematically maximized by correlation-oriented prompting alone. However, all generations amplify gender-related biases, especially in prosodic cues such as pitch and speech rate, and in facial behaviours such as smiling, blinking, and raised cheeks. Overall, the proposed tool facilitates the generation, evaluation, and critical analysis of multimodal interactions, contributing to bridging the gap between real and LLM-simulated data.