This work proposes FineMoLA, a weakly supervised framework that learns fine-grained frame--phrase correspondence directly from clip-level annotations, and efficiently infers pseudo frame-level alignments without human labeling.
Abstract
Text-conditioned human motion generation has made rapid progress with the emergence of large-scale motion--language datasets. However, even datasets with rich long-form descriptions typically provide supervision only at the clip level, without explicit temporal correspondence between motion frames and language. This limits fine-grained motion--text grounding and temporally precise generation. We propose FineMoLA, a weakly supervised framework that learns fine-grained frame--phrase correspondence directly from clip-level annotations. Our method first segments long-form descriptions into action-bearing phrases, and then formulates motion--language alignment as an optimal transport problem, which naturally models many-to-many relations between motion frames and text under global constraints. With entropic regularization and Sinkhorn iterations, FineMoLA efficiently infers pseudo frame-level alignments without human labeling. Experiments on SnapMoGen demonstrate that the learned alignments outperform baselines in motion--text grounding.
This work introduces ActionLMM, a memory-augmented vision-language model for long-video action summarization that aligns visual and motion modalities through joint representation learning and leverages a novel dual-memory mechanism to retain both local motion details and global temporal structure.
Rui-Rui Li, Dari Abdullah Alrwoaily, Turgut Sofuyev et al.· International Conference on...· 0 citations
Fine-grained video action recognition remains challenging because action categories often differ only in subtle inter-class variations and complex temporal dynamics. Recent Contrastive Language–Image Pre-training (CLIP)-based extensions perform well on general action recognition, but they typically rely on early global pooling of video features. Such coarse representations discard the fine temporal cues that distinguish subtle actions, causing a granularity mismatch in cross-modal alignment. To address this, we propose Structure-Aware Semantic-Adaptive (SASA)-CLIP, a framework for multi-granular cross-modal alignment. SASA-CLIP adopts a dual-branch design: a coarse-grained branch captures the global context, while a fine-grained branch matches descriptors against individual frames before aggregation, rather than pooling features early. To keep this alignment temporally coherent, we introduce a Gaussian prior as a temporal structural constraint, encoding the inductive bias of local temporal continuity into the attention matrix to guide an ordered alignment of key action segments along the temporal axis. On Kinetics-400 (ViT-B/32), SASA-CLIP reaches a Top-1 accuracy of 81.37%, improving over the X-CLIP baseline by 0.97%; on HMDB-51 and UCF-101 (ViT-B/16), it reaches 74.0% and 96.81%, improving by 3.25% and 2.61%, respectively. It also transfers to the zero-shot setting, improving over the baseline on HMDB-51 and UCF-101. These results show that combining multi-granular representations with a temporal structural prior benefits fine-grained recognition, suggesting that SASA-CLIP is a practical option for real-world visual sensing applications such as intelligent surveillance and wearable activity monitoring.
Xiaowei Han, Wenbao Si, Honghui Zhang et al.· Italian National Conference...· 0 citations
A cross-modal temporal alignment framework that combines a multi-scale temporal convolutional encoder with capsule-based dynamic routing, jointly optimizing temporal boundary prediction, cross-modal semantic alignment, and capsule diversity is proposed.
Gengtian Shi, Chenhao Wu, Shaofei Wang et al.· IEEE Access· 0 citations
This work proposes POP-TMR which pretrains body part representations for fine-grained text-motion retrieval and introduces HumanML3D+, an enhanced benchmark that provides accurate positive annotations for text queries and includes text descriptions at varying levels of detail, enabling more systematic performance assessment.
Kejun Lin, Shizhe Chen, Anwen Hu et al.· ACM Transactions on Multimed...· 0 citations
A closed-loop synthesis-and-verification pipeline that produces Omni-MoEdit, a large-scale dataset spanning body-part, amplitude, temporal, action, and style edits, and a unified latent flow-matching model that shares broad semantic and kinematic knowledge between generation and editing is developed.
Yilei Hua, Beibei Jing, Ce Zheng et al.· 0 citations
The Robust and Fine-grained training framework for CLIP-based vision-language models (RoFLIP) is proposed, enhancing both the robustness and granularity of vision-language alignment and underscore RoFLIP’s compositional reasoning and generalization abilities.
Yiwei Sun, Chuanbin Liu, Shancheng Fang et al.· International Journal of Com...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
