Skip to content
Conference Open access

Integrating Temporal Supervision and Self-Attention for Audio-Driven Head Synthesis

Aug 2026 · International Conference on Multimedia Analysis and Pattern Recognition · pp. 622-627 · 0 citations · 40 references

TL;DR

TemporalTalk is presented, which closes this gap with three training-time mechanisms: a face-masked temporal consistency loss with stop-gradient and delayed activation, a multi-head self-attention module that replaces the fixed convolutional attention over the audio context window, and cosine learning-rate annealing with linear warmup.

Abstract

NeRF-based talking head methods can render individual frames with impressive fidelity, yet the assembled videos often flicker. The reason is structural: each frame is optimized independently, so nothing in the training objective ties frame t to frame t − 1. We present TemporalTalk, which closes this gap with three training-time mechanisms: a face-masked temporal consistency loss with stop-gradient and delayed activation, a multi-head self-attention module that replaces the fixed convolutional attention over the audio context window, and cosine learning-rate annealing with linear warmup. None of the three touches the inference path, so runtime cost is unchanged. On the May benchmark, TemporalTalk reduces temporal flicker measured by tLPIPS (the mean Learned Perceptual Image Patch Similarity between consecutive generated frames) by 24% relative to SyncTalk, while also improving PSNR by 0.14 dB, LPIPS (Learned Perceptual Image Patch Similarity) by 3.0%, and landmark distance by 2.3%. An ablation study shows that all three mechanisms contribute, with the temporal loss accounting for the largest share of the gain.

Read PDF

Similar papers

Book Open access Nov 2026

LocTFL: Self-Supervised Coarse-to-Fine Audio-Visual Temporal Localization of Local Deepfakes

Local audio-visual deepfakes modify only a short word or phrase within an otherwise authentic video. Because the manipulated evidence occupies a small temporal region, global aggregation weakens detection cues, while accurate start/end estimation becomes as important as video-level classification. We present LocTFL, a...

Hua-Ye Wang, Bo Wu, Long-Teng Jiang et al. · 0 citations
Sep 2026

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

TT-VidT leads Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously, improving over the strongest non-TT row by 54% ~ 121%, while using 48% fewer encoder FLOPs than DisMo and 55% fewer than VideoMAE or V-JEPA2.

Shih-Ying Yeh, Daniel Z. Kaplan, Xue-Hai Wang et al. · 0 citations
Open access Oct 2026

PAGE: Parts-Aware GuidancE for Co-Speech Gesture Portrait Video Generation

Co-speech gesture video generation aims to synthesize realistic, audio-driven portrait videos from a single reference image and a speech signal. Despite advances in visual quality, diffusion-based approaches still suffer from degraded local details and imperfect audio-visual alignment. We present PAGE (Parts-Aware Guid...

Ya-Biao Wang, Jia-Fu Wu, Y. Xu et al. · 0 citations
Open access Nov 2026

An efficient spatial-temporal attention network for low-latency isolated Indian Sign Language Recognition

About 63 million people in India live with significant hearing impairment, yet Indian Sign Language (ISL) is still poorly served by assistive digital tools even though it is central to communication within the Deaf and Hard-of-Hearing (DHH) community. We present STANet, a compact hybrid model for isolated ISL recogniti...

Surbhi Maheshwari, Amit Sinhal, Arpana Sinhal · 0 citations
Oct 2026

Exploring Multimodal Adapters with Multi-Task Decoders for Video Action Recognition.

Large-scale vision-language pretrained models like CLIP, paired with parameter-efficient fine-tuning techniques, have emerged as promising solutions for image-to-video transfer in video action recognition. However, existing methods often prioritize strong supervised performance at the expense of transferability and gen...

Meng-Meng Wang, Ze-Yi Huang, Bo-Yuan Jiang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.