Text-to-motion generation must produce motions that are semantically correct, temporally coherent, and physically plausible. A natural approach is to first project motion data into a structured semantic space and then train a generative model within that space. Such a paradigm has been highly successful in image generation through Representation Autoencoders (RAEs), where a frozen self-supervised encoder provides semantic features for diffusion or flow models to learn from. However, direct transfer of such a paradigm to motion space using Motion-JEPA as the frozen encoder fails dramatically. We diagnose this failure geometrically and identify two motion-specific bottlenecks: (1) the JEPA feature space is spectrally ill-conditioned, making the Gaussian-to-data transport unstable; and (2) even with a well-conditioned spectrum, flow residuals tend to align with decoder-sensitive directions, where small latent errors are amplified into large motion artifacts after decoding. Based on these insights, we propose MoRAE. MoRAE addresses the two bottlenecks separately. A compact bottleneck distills the structured JEPA representation while removing weak and redundant directions, bringing the latent spectrum into a transport-stable regime. Motion-coupled training then aligns the retained latent geometry with the decoder, making characteristic flow errors less costly after decoding. With this flow-friendly latent, a standard non-autoregressive Flow-Matching DiT achieves state-of-the-art performance.
Text-to-motion generation aims to synthesize semantically consistent and naturally coherent motion sequences from natural language descriptions. Given the continuous nature of human motion, diffusion models operating in a continuous latent space offer inherent advantages over vector quantization-based methods, particularly in avoiding quantization errors and in modeling quality. However, existing diffusion models primarily rely on mean squared error loss. This stepwise regression paradigm often leads to ‘over-smoothed’ motion sequences and struggles to capture the subtle semantic nuances embedded in textual descriptions. To realize the potential for continuous diffusion generation, an enhanced latent-space diffusion framework designed to elevate generation capabilities across two dimensions, namely, distribution approximation and semantic alignment, is proposed. Specifically, a latent-space adversarial discriminator is incorporated. By applying decoupled adversarial supervision, this component mitigates the detail loss caused by mean regression, significantly enhancing the physical realism and dynamic sharpness. Concurrently, a latent-space contrastive alignment strategy is introduced during the denoising process that reinforces the correspondence of the generated motion sequences with the given textual inputs via explicit cross-modal constraints. Extensive experiments on standard benchmarks demonstrate that the proposed method effectively addresses the limitations of conventional diffusion models, thus validating the potential of continuous diffusion frameworks within the domain of text-driven motion synthesis.
Zhaowu Li, Rui Liu, Deheng Zhu et al.· Visual Computing for Industr...· 0 citations
Recent advances in diffusion models and Transformer architectures have led to significant progress in text-to-video generation. However, these models often suffer from semantic errors such as missing objects, incorrect attributes, or mismatched actions. Although some semantic correction methods perform optimization before sampling or refinement after sampling, how to detect and correct semantic deviations during the video generation process remains underexplored. In this paper, we introduce a training-free, interpretable mid-generation correction framework that integrates multimodal large language model (MLLM) feedback directly into the diffusion sampling loop. Our framework achieves diffusion trajectory correction by injecting semantic evaluation signals during video synthesis, enabling the model to optimize the generated content through continuous self-reflection. We propose two key modules: a Semantic Assessment Supervisor that generates intermediate preview frames for semantic evaluations and deviation diagnostics, and a Semantic Modification Assistant that corrects semantic drift during inference via a controllable latent trajectory intervention. Our method improves semantic alignment, visual fidelity, and temporal consistency without modifying model parameters. We validate the effectiveness of our approach through extensive experiments across multiple benchmarks.
Junhao Chen, Zheqi Lv, Keting Yin et al.· 0 citations
Zero-shot skeleton-based action recognition (ZSAR) aims to recognize unseen action categories by aligning skeleton features with textual semantics. However, existing methods rely on text-derived prototypes that inherently lack geometric structure and physical constraints, resulting in a pronounced \textit{semantic-kinematic gap}. To bridge this gap, we propose \textbf{GenPrior}, the first framework to exploit generative priors from pre-trained Text-to-Motion (T2M) models for ZSAR. Specifically, we introduce Dispersion-Gated Feature Fusion, which distills kinematic prototypes and intra-class dispersion from generative motion sequences and employs a learned gating network to adaptively inject reliable structural cues into textual embeddings while suppressing synthetic artifacts. Furthermore, we propose Generative Prototype Refinement, which leverages these generation-enhanced prototypes as anchors to mine high-confidence unseen samples, calibrating class prototypes toward the true distribution and thereby unleashing strong performance gains. Extensive experiments on NTU-60, NTU-120, and PKU-MMD demonstrate that GenPrior achieves state-of-the-art performance under both zero-shot and generalized zero-shot settings. Code is available at https://github.com/jidongkuang/GenPrior.
Methods for text-based generation of hand-object interaction (HOI) sequences primarily focus on producing smooth, physically plausible trajectories. A truly utilitarian method should additionally support variable-length generation, composite motion sequences, motion completion and infilling, and reliable termination without compromising physical plausibility. Standard diffusion models for HOI generation are typically trained only for text-to-motion generation on atomic motions and require the motion length to be specified a-priori. Autoregressive (AR) methods provide greater sequence-level flexibility, but commonly depend on discrete motion codes, which can lose contact-sensitive motion detail. To address these key limitations, we present a model performing Masked Autoregression with Diffusion for HOI generation (MAD-HOI). Our method starts by encoding hand and object motions in a continuous latent space while keeping them disentangled to maintain stream-wise control. This is followed by a masked autoregressive transformer to predict context features that condition a flow-matching head. MAD-HOI is capable of motion generation for atomic and composite articulated sequences, conditioned motion completion and infilling, as well as EOM (End of Motion) prediction from a single training objective. We provide comprehensive evaluations for these capabilities and benchmark our method on the ARCTIC and GRAB datasets. Our experiments demonstrate that our method generates more diverse and physically plausible interactions compared to other open-sourced baseline methods.
Ananya Bal, Kartik Sharma, E. Lai et al.· 0 citations
Text-to-Audio-Video (T2AV) generation aims to produce temporally and semantically aligned visual and auditory content from natural language descriptions. While recent progress in text-to-audio and text-to-video models has improved generation quality within each modality, jointly modeling them remains challenging due to incomplete and asymmetric correspondence: audio often reflects only a subset of the visual scene, and vice versa. Naively enforcing full alignment introduces semantic noise and temporal mismatches. To address this, we propose a novel framework that performs selective cross-modal alignment through a learnable masking mechanism, enabling the model to isolate and align only the shared latent components relevant to both modalities. This mechanism is integrated into an adaptation module that interfaces with pretrained encoders and decoders from latent video and audio diffusion models, preserving their generative capacity with reduced training overhead. Theoretically, we show that our masked objective provably recovers the minimal set of shared latent variables across modalities. Empirically, our method achieves state-of-the-art performance on standard T2AV benchmarks, demonstrating significant improvements in audiovisual synchronization and semantic consistency.
Jiyang Zheng, Siqi Pan, Yu Yao et al.· Neural Information Processin...· 6 citations
Grounding human motion in language, and language in motion, is a central step toward physical AI systems that can understand, generate, and communicate human behavior. Unified motion--language systems first coupled the two directions through a shared discrete motion codebook, but quantization limits generation quality. The strongest generators buy quality back at growing cost: stacked residual codebooks enlarge the representation; masked decoding stages, long autoregressive rollouts, and denoising chains of tens to hundreds of steps stretch inference; even the continuous-latent designs among them reach their latent only through an iterative diffusion head; and none of this decoding machinery serves understanding. We therefore propose MUGEN, a unified motion--language framework that pays neither cost: no codebook, one draw. A single adaptive-length autoencoder compresses any-length motion into a few continuous latent slots, the system's only motion representation: the language model generates them for text-to-motion and reads them back for motion understanding. Depth-routed hidden states let each slot read from the transformer depth it needs, and a calibrated head predicts a joint distribution over the full latent set, so a single draw carries the text-conditional, cross-slot variation a description permits. At a decoding cost of K language-model steps, one draw, and one decoder pass, MUGEN leads language-model baselines on FID on HumanML3D while raising retrieval precision above the real-motion reference under the standard evaluator, achieves the best CIDEr and BLEU@4 scores, and surpasses the discrete-token state of the art on every retrieval and alignment metric on SnapMoGen.
Zhankai Ye, Yukai Jin, Bingyang Wei et al.· 0 citations