Skip to content

Scaffold Then Internalize: Representation Injection for Diffusion Transformers

Sep 2026 · arXiv.org · Vol abs/2609.35292 · 0 citations · 57 references
Computer Science

TL;DR

This work introduces REPI, a training framework based on a scaffold-to-internalization strategy, in which projected encoder representations initially serve as a temporary scaffold and are then progressively internalized by the diffusion transformer.

Abstract

Recent representation alignment (REPA) methods accelerate diffusion transformer training by aligning projections of the transformer's hidden states with representations from pretrained visual encoders. In this work, we explore a reverse and complementary direction to REPA: rather than projecting diffusion representations into the encoder's space, we inject encoder representations into the diffusion transformer, allowing them to actively participate in the denoising process. To this end, we introduce \textit{REPresentation Injection} (REPI), a training framework based on a scaffold-to-internalization strategy, in which projected encoder representations initially serve as a temporary scaffold and are then progressively internalized by the diffusion transformer. REPI outperforms REPA across a wide range of backbones and is highly complementary to it: combining the two yields substantial gains over either alone. Notably, with only 160K training steps, REPI + REPA matches vanilla SiT trained for 7M steps, a speedup of over $43.5\times$. Code will be available at https://jeneveuxpas.github.io/REPI

View source

Similar papers

Preprint Oct 2026

Representation Conditioning for Generation

Representation-alignment methods effectively accelerate the diffusion transformer training. REPA, for example, uses a pretrained vision encoder to supervise an intermediate layer through an MLP projection head which is discarded after training. It aims to import semantics from the vision encoder to the diffusion backbo...

S. Razavi, Xing-Jian Leng, Liang Zheng · 0 citations
Preprint Oct 2026

Learning Representations to Guide Pixel Diffusion Transformers

Visual representations are largely developed for understanding, yet what makes them useful for generation remains unclear. We study this question in pixel diffusion transformers and uncover two key insights. First, better representation quality does not necessarily translate into greater generative utility, as external...

Zi-Hao Pan, Meng-Ting Chen, Cheng Shi et al. · 0 citations
Preprint Sep 2026

RepFlow: Reciprocal Supervision Improves Generation and Representation in Flow Models

Generative models learn visual structure through denoising, yet their internal states are entangled with both noise level and network depth, making it difficult to obtain a stable visual representation from the generator itself. We introduce RepFlow, which learns such a representation from the generator's evolving comp...

Wei-Li Zeng, Feng Tian, Sheng-Qi Liu et al. · 0 citations
Preprint Oct 2026

CoVA: Composing VAE Representations for Generative Learning

CoVA is introduced, a lightweight framework that composes VAE features online into stage-wise guidance for flow-matching training, and uncovers a hidden confound in soft label conditioning: averaging class embeddings attenuates their magnitude, coupling semantic interpolation with conditioning strength.

Zhao-Wei Chen, Yi-Kai Ge, Lin-Ze Li et al. · 0 citations
Preprint Sep 2026

PixelDense: Dense Prediction as Representation Alignment for Pixel Diffusion

PixelDense is introduced, which routes DINOv2 and SAM2 through a semantic projection stream, routes Depth Anything v2 and Metric3D v2 through a geometric projection stream, and adds a weight-space orthogonality penalty that keeps the two streams in disjoint subspaces.

Lehan Yang, Dai-Qing Qi, Wen-Hao Zhang et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Oct 28, 2026

Making machines that can listen

Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.

Microsoft Research Blog Aug 11, 2026

Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

Radiology AI is evolving beyond report generation. CARE-X explores a unified approach that combines flexible reasoning, calibrated predictions, and measurement-based tools for chest X-ray interpretation. The post Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.