TDDN: Text-aligned Diffused DINO Network for Puzzle Understanding
TL;DR
This work fuse DINOv3 and CleanDIFT representations into a perception encoder and align it with RoBERTa-L, yielding a text-aligned model TDDN that preserves this perceptual advantage and leads on segmentation benchmarks among general-purpose contrastive encoders, including SigLIP.
Abstract
Structured visual reasoning, such as image puzzles, demands fine-grained visual perception, which current Vision Language Models (VLMs) often lack. VLMs built on CLIP-based ViT backbones tend to trade fine-grained visual details for high-level semantics. We show that this deficit propagates downstream, affecting various vision and language tasks. To address this, we fuse DINOv3 and CleanDIFT representations into a perception encoder (DiffusedDINO), and align it with RoBERTa-L, yielding a text-aligned model (TDDN) that preserves this perceptual advantage: with frozen backbones and only ~590K alignment pairs, TDDN surpasses CLIP ViT-L/14 in image-text retrieval tasks: text-to-image on Flickr30K and both directions on MS-COCO-2014. TDDN more than triples CLIP's dense-prediction accuracy (ADE20K 5.20 ->18.11 mIoU, COCO-Stuff 7.35 ->24.44), despite CLIP's massive training corpus. TDDN leads on segmentation benchmarks among general-purpose contrastive encoders, including SigLIP 2. We further introduce PuzzlePerception, a segmentation and visual question answering dataset that probes fine-grained spatial understanding, on which TDDN doubles CLIP's segmentation accuracy (11.04 ->22.51 mIoU). Our code and data are available at https://github.com/adityaSomak/TDDN.
