Twin, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so the sequence length is unchanged and attention cost does not increase is proposed, which performs competitively on multimodal understanding benchmarks and improves reconstruction fidelity.
Abstract
Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation. Discrete methods unify the interface via a shared codebook, whereas continuous pipelines often rely on two disparate representations -- semantic features (e.g., ViT) for understanding and low-level latents (e.g., VAE) for synthesis -- resulting in mismatched latent spaces. We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so the sequence length is unchanged and attention cost does not increase. However, jointly modeling Twins in a Diffusion Transformer exposes a severe optimization imbalance: the model fits the ViT component well but struggles to match the VAE latent distribution. We trace this imbalance to three sources of heterogeneity: frequency bias, intrinsic dimensionality, and condition-aligned vs condition-independent uncertainty. To address it, we adapt a focal regression objective for flow matching that upweights large-error VAE dimensions, better balancing optimization across the ViT and VAE components. On ImageNet, this yields up to 10.57 gFID gain over naive MSE loss without classifier-free guidance. Twins also performs competitively on multimodal understanding benchmarks and improves reconstruction fidelity, narrowing the gap between understanding- and generation-oriented representations.
Multimodal models often align language, vision, and audio in a single final-layer latent space, implicitly assuming that task-relevant evidence emerges at the same semantic depth across modalities. Using layer-wise CKA analysis, we observe that this assumption leads to semantic granularity mismatch: textual cues usually require deeper contextual abstraction, whereas visual and acoustic cues often provide discriminative perceptual evidence in shallow or middle layers. This mismatch can flatten fine-grained modality-private cues and reduce reliability under noisy, imbalanced, or missing inputs. To address this, we proposed Adaptive Hierarchical Representation Alliance (AHRA), a hierarchical shared--private expert framework. AHRA factorizes each modality into shared and private streams across semantic levels, regularizes them with shared alignment and private decorrelation, routes shared information through a cross-modal expert, and enhances task-relevant private tokens with modality-specific experts guided by a sparsity-controlled soft-gating mechanism (foreground exam). A hierarchical co-fusion module then performs intra-level expert coordination and inter-level semantic selection. Experiments on six benchmarks across image-text classification, multimodal intent recognition, and trimodal sentiment analysis show that AHRA consistently improves over strong baselines and remains robust under noisy and missing-modality settings.
Chunlei Meng, Pengbin Feng, Jacqueline J. Pang et al.· 1 citation
It is shown that steering vectors learned from the understanding branch can transfer to generation, enabling controllable image synthesis and improved semantic faithfulness, and establish cross-branch steering as a practical tool for probing multimodal representations.
This work studies raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model and shows that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior.
Marcel Plocher, Bernhard Schölkopf, Andreas Geiger et al.· 0 citations
SPAE employs a compact bottleneck to distill stable semantic information while suppressing high-frequency components, thereby improving the alignment between DiT-generated latents and encoder latents, and achieves a favorable balance among visual understanding, generation quality, and reconstruction fidelity.
Yibin Huang, Ji-Xiang Hong, Zongzhao Li et al.· 0 citations
Joint-Embedding Predictive Architectures (JEPAs) have emerged as a principled framework for self-supervised learning of world models in compact latent spaces, yet existing methods are fragmented: some predict masked parts of a single image in latent space (I-JEPA), others learn to predict global photometric transformations (Image World Models), while video-scale JEPAs predict future temporal states and are post-trained for action-conditioned planning (V-JEPA~2, DINO-World, DINO-WM). These objectives are treated as distinct recipes with separate encoders, predictors, and anti-collapse regularizers, hindering a single model from unifying image-level and video-level world modeling. We present UniJEPA, a unified JEPA that jointly learns photometric prediction (image-level transformations) and temporal prediction (video-level next-state dynamics) in one shared latent space. A single end-to-end objective, composed of a next-embedding prediction loss and a Gaussian regularizer, yields a provably anti-collapse encoder-predictor pair trainable from raw pixels without EMA, stop-gradient, or pre-trained encoders. We show that the same latent space supports controllable abstraction: photometric prediction learns invariant structure while temporal prediction learns equivariant dynamics. After action-conditioned post-training on offline trajectories, UniJEPA enables zero-shot planning by treating goal features as prediction targets. On image, video, and control benchmarks, UniJEPA matches or surpasses task-specific JEPAs while requiring a single loss hyperparameter, and plans up to tens of times faster than generative world models at comparable accuracy.
Andriana Lanji, Dawei Liu, Jin Li et al.· 0 citations
Generating a virtual anchor from speech has to satisfy three demands at once: the face must stay recognizable as the same person, the motion has to be temporally coherent, and the lips must follow the audio. Existing methods often fall short on the first two points because identity and expression share one entangled representation, and temporal dynamics are modeled only implicitly. We propose DynaID-VAE to address these problems. At its core is an identity–expression disentangled conditional VAE (DC-VAE) that splits the latent space into a time-varying expression subspace and a static identity subspace, held apart by mutual-information minimization and orthogonality regularization. A temporal memory module (TMM) then regularizes the expression trajectory: a GRU propagates sequential state, attention retrieves from a learnable key–value prototype memory, and residual fusion combines the two. Multiscale adversarial supervision and lip–audio synchronization losses complete the training objective. We evaluate on VirtualAnchor-100, a benchmark we recorded ourselves (100 h, 10 anchors), under two complementary protocols. Cross-identity driving is scored only with non-paired measures, namely lip synchronization, distributional video quality, and identity preservation; full-reference image metrics are confined to a self-reenactment protocol, where a genuine paired ground truth exists. DynaID-VAE outperforms the one-reference baselines Wav2Lip, PC-AVS, SadTalker, and DiffTalk under both protocols and on unseen VoxCeleb2 identities. The margins are stable across five identity-disjoint, nested cross-validation folds and are confirmed by an external SyncNet evaluator that never takes part in training, while the model runs at 41.2 FPS with 14.3 M parameters. Ablations separate the contribution of each regularizer and each TMM component. Linear and capacity-matched non-linear probes quantify the factorization as a large reduction of decodable reference identity; full independence is not claimed. A user study confirms the perceptual gains.
Unknown authors· Electronics· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
