Skip to content

Generative Embedding Benchmark: How Much Information Survives in a Dense Embedding?

Aug 2026 · arXiv.org · Vol abs/2608.06972 · 0 citations · 29 references
Computer Science

TL;DR

The Generative Embedding Benchmark (GEB) is introduced, in which a decoder answers questions using only a frozen embedding and question text, without access to the original image or intermediate visual features, showing that generative readout exposes information bottlenecks that separability-based evaluation does not capture.

Abstract

Embeddings have emerged as a standard representational interface linking foundation models with downstream systems. Most embedding benchmarks assess representations through discriminative tasks or geometric criteria centered on separability in embedding space. However, strong performance on such evaluations does not establish whether content compressed into an embedding remains accessible to a downstream generator. To address this gap, we introduce the Generative Embedding Benchmark (GEB), in which a decoder answers questions using only a frozen embedding and question text, without access to the original image or intermediate visual features. Answer quality under this readout measures generative information: the answer-relevant content recoverable from an embedding. GEB includes a curated visual-question-answering dataset with a 1,800-item development split and a held-out 900-item test split covering natural images, scene text, and visual documents. Using a common decoder and training recipe, we evaluate seven public embedding models in visual-only and vision-language joint modes. On the test set, visual-only scores range from 28.25 to 33.21; with image-question joint encoding, all five VLM-based embedding models score higher, and the best reaches 65.56. Matched embeddings also outperform text-only inputs, zero embeddings, and shuffled embeddings. Natural-image information is much easier to recover than scene text or visual-document information, while a Qwen3-VL-2B reference with access to the original image reaches 84.30. Together, these results show that generative readout exposes information bottlenecks that separability-based evaluation does not capture.

View source

Similar papers

Conference Open access 2026

Direct Projection of Visual Features into Linguistic Embeddings for Domain-Adapted Multimodal Learning

This paper introduces a novel multimodal learning framework that directly projects visual features into linguistic embeddings, enabling robust, domain-adapted object recognition across diverse categories. Leveraging state-of-the-art visual (Detectron2 Mask R-CNN) and language (TinyLlama) models, the architecture incorp...

J. Kralev · 0 citations
Preprint Nov 2026

Factorized MLLM Embeddings for Composed Image Retrieval

Composed Image Retrieval (CIR) aims to retrieve a target image given a reference image and a modification text. Although recent approaches leveraging Multimodal Large Language Models (MLLMs) have shown significant progress, we argue that the MLLM's compositional understanding remains underutilized. This stems from dire...

WonJun Moon, Seongchan Kim, Dahun Kim et al. · 0 citations
Preprint Oct 2026

CoVA: Composing VAE Representations for Generative Learning

CoVA is introduced, a lightweight framework that composes VAE features online into stage-wise guidance for flow-matching training, and uncovers a hidden confound in soft label conditioning: averaging class embeddings attenuates their magnitude, coupling semantic interpolation with conditioning strength.

Zhao-Wei Chen, Yi-Kai Ge, Lin-Ze Li et al. · 0 citations
Open access Nov 2026

How Much Do Modern Embeddings Reveal? Inversion Attacks on Commercial Embedding Models

Text embedding inversion reconstructs source text from fixed-dimensional vectors, posing a privacy risk to systems that store embeddings instead of raw text. We adapt the vec2text framework (Morris et al., 2023) to support text-embedding-3-small and mistral-embed alongside GTR-base. We evaluate 800 samples per category...

Kareem Elaasar, Cristina V. Lopes · 0 citations
Preprint Oct 2026

Lost in the Latent Space: How Code Embeddings Represent Functional Intent and Implementation Fidelity

Code embedding models have become a key building block for many software engineering applications, yet their internal representations remain poorly understood. Existing evaluations tend to focus on downstream task performance, offering limited insight into whether a single embedding can support both functional abstract...

Lu Sun, Yu-Heng Huang, Lei Ma et al. · 0 citations
#natural language process... Preprint Oct 2026

More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training

Native multimodal pre-training trains a single model from scratch on text and multimodal data, enabling it to understand both language and visual inputs. However, how visual signals shape scaling behavior and support capabilities beyond visual understanding remains unclear. We investigate this question through controll...

Haoyuan Wu, Ao-Qi Wu, Jia-Ni Hu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.