Redemption Score: A Multi-Modal Evaluation Framework for Image Captioning via Distributional, Perceptual, and Linguistic Signal Triangulation
TL;DR
Redemption Score is introduced, a novel evaluation framework for multi-modal generation by triangulating three complementary signals: Mutual Information Divergence for global image-text distributional alignment, DINO-based perceptual similarity of cycle-generated images for visual grounding, and LLM Text Embeddings for contextual text similarity against human references.
Abstract
Evaluating image captions requires cohesive assessment of both visual semantics and language pragmatics, which is often not entirely captured by most metrics. As such metrics increasingly guide model development, benchmarking, and system optimization in multimodal AI, inaccuracies in evaluation can misrepresent true progress. We introduce Redemption Score(RS), a novel evaluation framework for multi-modal generation by triangulating three complementary signals: (1) Mutual Information Divergence (MID) for global image-text distributional alignment, (2) DINO-based perceptual similarity of cycle-generated images for visual grounding, and (3) LLM Text Embeddings for contextual text similarity against human references. A calibrated fusion of these signals allows RS to offer a more holistic assessment. On the Flickr8k benchmark, RS achieves a Kendall-$\tau$ of 58.42, outperforming most prior methods and demonstrating superior correlation with human judgments without requiring task-specific training. Our framework provides a more robust and nuanced evaluation by thoroughly examining both the visual accuracy and text quality together, with consistent performance across Conceptual Captions and MS COCO.
