ENCORE, an Event-Assisted Complementary Motion Refinement framework for learned video compression, employs Complementary Motion Representation to decompose aligned RGB-event features into common and modality-specific motion representations and identifies event-specific responses that are active and novel relative to RGB.
Abstract
Learned video compression relies on accurate temporal modeling to remove redundancy between adjacent frames. However, most existing codecs infer motion solely from discretely sampled RGB frames, making their estimates vulnerable to fast motion, blur, occlusion, weak texture, low illumination, and abrupt brightness changes. Event cameras asynchronously capture fine-grained intensity changes between RGB timestamps and therefore provide complementary evidence about inter-frame dynamics. We propose ENCORE, an Event-Assisted Complementary Motion Refinement framework for learned video compression. ENCORE first employs Complementary Motion Representation (CMR) to decompose aligned RGB-event features into common and modality-specific motion representations. Spatial Energy and Redundancy-Informed Calibration (SERIC) then identifies event-specific responses that are active and novel relative to RGB, suppresses weak or redundant evidence, and predicts a candidate flow correction. Finally, Energy-Aware Routing (EAR) determines where and how strongly the correction should refine the RGB flow. Events serve solely as an auxiliary modality for motion modeling, while RGB remains the only coding and reconstruction target. Experiments on BS-ERGB, HQ-EVFI, and CED demonstrate consistent gains across datasets and GOP lengths. On BS-ERGB, ENCORE achieves up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings, while retaining clear improvements on the other two datasets.
This work proposes an adapter-based framework that incorporates event-derived cues into a pre-trained image-to-video diffusion model with minimal architectural changes and consistently outperforms existing state-of-the-art approaches.
Guixu Lin, Yuyang Yu, Xiang Ji et al.· 0 citations
Event-guided video super-resolution (VSR) leverages high-temporal-resolution event streams to address motion blur, rapid dynamics, and poor illumination that challenge frame-only VSR methods. However, most existing approaches emphasize reconstruction quality while overlooking real-time performance and computational efficiency, limiting their deployment in latency-sensitive scenarios. To overcome these issues, we present E2VSR, a lightweight and Efficient Event-guided VSR framework tailored for real-time applications. Operating under a causal setting with only current and past observations, E2VSR is designed for low-latency event-guided VSR. We propose an event-confidence adaptive propagation strategy comprising two key modules: the Event-induced Feature Modulation (EvFM) block for robust cross-modal event-frame integration, and the Event-Confidence Feature Fusion (EvCFF) block, which exploits events as motion cues for adaptive inter-frame aggregation. This design improves motion-aware temporal aggregation in challenging dynamic conditions, where event cues may provide complementary temporal information. Furthermore, an Implicit Event Reconstruction (IER) technique leverages event information during training to enrich feature representations without adding inference-time cost, enhancing spatial and temporal fidelity. Experimental results demonstrate that E2VSR achieves superior quantitative and qualitative performance while maintaining a low parameter count and computational cost.
MotionCraft is presented, a controllable VSR framework that formulates restoration as motion-aware latent state prediction inspired by world models and integrates adaptive sparse attention with an explicit user-accessible control interface to deliver temporally consistent, high-quality reconstructions under streaming constraints.
Rong Fu, Chunlei Meng, Yangcheng Zeng et al.· 0 citations
This novel E-VFI framework diverges from approaches reliant on direct image-level supervision by constructing multilevel, degradation-insensitive semantic perceptual supervisory signals to enhance the perceptual realism and multi-scene generalization of the model's predictions.
Yuhan Liu, Linghui Fu, Zheng Yang et al.· Neural Information Processin...· 1 citation
Understanding long-range videos remains a key challenge in computer vision due to high temporal redundancy and computational burden. Despite strong performance of recent models, they are constrained in terms of scalability and generalization when applied to longer video sequences. In this work, we present Keyframe-based Spatio-Temporal Adaptive Representation (K-STAR), a redundancy-aware video summarization framework designed to generate compact and semantically rich representations that are effective in downstream tasks. The proposed method jointly models appearance and motion cues while filtering redundant frames. Importantly, it preserves critical temporal transitions while significantly reducing the number of processed frames. Additionally, each key frame is encoded using object, scene, and background-aware prompts, enabling richer semantic representation. Evaluated on the UCF-101 dataset, K-STAR achieves Top-1 accuracy of 93.06% and Top-5 accuracy of $\mathbf{9 8. 7 3 \%}$, with $\mathbf{5 6} \times$ frame reduction and $\mathbf{1 1. 5} \times$ faster inference, demonstrating competitive performance with substantially improved efficiency.
Rahul Kumar, S. Channappayya· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
