Skip to content

BindCLIP: One Balanced Coupling For Compositional Vision Language Scoring

Sep 2026 · arXiv.org · Vol abs/2609.23717 · 0 citations · 23 references
Computer Science

Abstract

Global vision--language similarities compress an image and a caption into one vector, preserving semantics but not which word corresponds to which region or how those regions are arranged; a model can recognize every word and object yet prefer a compositionally incorrect caption. We argue that a frozen encoder retains this association structure, so the problem is to read it rather than to rebuild it beside the pretrained similarity. We introduce BindCLIP, a pairwise scorer built on one latent object: a balanced token--patch--depth optimal-transport coupling that places both candidate captions and several visual depths in a single plan. Semantic, entity, order, and spatial evidence are read as energies of this state, and exchanging the candidates permutes the plan, making the score exactly antisymmetric. A geometric refinement inside the coupling contracts moves that the candidates and the visual depths do not support. No task label, parser, relation inventory, or detector is used. One checkpoint and one inference path improve the official What'sUp, ARO, and SugarCrepe benchmarks over frozen global CLIP, with the strongest transfer on the relation splits. Controls rule out patch access and caption-length shortcuts, and an inference-time lesion localizes spatial arrangement to the coupling.

View source

Similar papers

Preprint Oct 2026

CompoSET: A Controlled Single-Edit Benchmark for Vision-Language Compositionality

Vision-language compositionality is the ability to identify which properties, relations, quantities, and states bind to which entities in both an image and a caption, and to match those bindings across modalities. Existing benchmarks evaluate this ability through image-text matching: a model ranks candidate image-capti...

Laure Delisle, Pietro Perona · 0 citations
Preprint Oct 2026

GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport

A child learns which words name which things from roughly 100 million words of input. Vision-language models trained at a comparable scale bind words to referents far less reliably, and we argue that part of the gap is structural: the usual next-token captioning objective has no variable standing for"this word refers t...

Rajdeep Pal, Jotiraditya Banerjee, Saptarshi Pani · 0 citations
Preprint Oct 2026

Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models

Retrieval is a fundamental task in multimodal systems: given an image or text query, the goal is to find the most relevant images or texts in a collection, often using a shared embedding space. Reliable retrieval requires these embeddings to capture not only which concepts are present, but also how they are composed. O...

Netanel Y. Tamir, Michael Green, Issar Tzachor et al. · 0 citations
Preprint Oct 2026

ALPS: Visual State Encoders for World Models

World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typ...

Yan-Qing Liu, Le-Tian Zhang, Cheng-Zhi Liu et al. · 0 citations
Preprint Sep 2026

Modern GLOM: Canonical locks that encode part-whole hierarchies

One of the challenges in representational learning is how to encode part-whole hierarchies in a neural net. Prior works rely on flattening tree-like structures into string-like sequences and training a sequence-to-sequence model via autoregression. While such a representation works for parse-trees in NLP, it is not ent...

Rajat Modi, Y. Rawat · 0 citations
Preprint Oct 2026

SheafPrune: Sheaf-Guided Graph Propagation for Visual Token Pruning

Multimodal large language models process the complete visual token sequence, increasing prefill cost even when many tokens are irrelevant to the instruction. Token-wise saliency captures limited inter-token dependence; graph-based selection instead depends on how relations govern importance propagation and ranking. We...

Xiao-Yang Ma, Chi Wang, Hong Sun · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.