ComedyQA: Revealing Visual, Cultural, and Reasoning Challenges in Video Humor Understanding
Abstract
Understanding humor in videos remains an underexplored challenge for Multimodal Large Language Models (MLLMs). It demands the integration of fine-grained multimodal perception, temporal narrative reasoning, and sociocultural knowledge, which existing works have not fully explored. Consequently, the absence of high-quality, targeted datasets has severely hindered quantitative evaluation and model development in this domain. To address this gap, we introduce ComedyQA, a comprehensive benchmark for video humor understanding. It comprises 2,526 comedy video clips and 12.6K fine-grained annotations covering multimodal humor cues, relevant plot context, external knowledge, and comprehensive humor explanations. It supports three tasks of increasing complexity: (1) Multimodal Humor Classification, (2) Multimodal Humor Grounding, and (3) Comprehensive Humor Explanation. Evaluations show that recent MLLMs still struggle with comprehensive video humor understanding. Our analysis identifies three key failure modes: visual humor misattribution, cultural unawareness, and contextual counter-intuitive reasoning deficits. Motivated by these findings, we propose VideoHumorAgent, a multi-agent framework that addresses each failure mode with specialized agents. Experiments demonstrate consistent improvements across all tested models. Our dataset and codes are released at https://github.com/wangxueyan666/ComedyQA.