The proposed HiResNets learn to foveate around scenes similarly to human vision, and have superior performance in difficult egocentric video recognition tasks, especially egocentric video with small objects and fine-grained recognition.
Abstract
Much of the recent progress in image and video recognition has come at the cost of memory: larger models, increased resolution, and longer temporal contexts. An inevitable component is the quadratic (or larger) growth of memory and compute based on image resolution, which is a property of the grid sampling used in convolutional networks and vision transformers. In this work we study residual networks whose convolutional blocks have logarithmic-square growth instead, enabling them to process very high-resolution video quickly. The key insight is to use a residual architecture's residual stream as a high-resolution buffer, to which convolutional blocks only read and write via log-polar image warp operations. Layers adaptively focus on different parts of each frame, with very high resolution only near the focus point. A complete high-resolution representation is built up in the residual stream, analogous to eye saccades creating a complete picture in biological vision, and a theoretical construction is presented that eliminates the quadratic dependency of the residual stream resolution. Experiments demonstrate that our proposed HiResNets learn to foveate around scenes similarly to human vision, and have superior performance in difficult egocentric video recognition tasks, especially egocentric video with small objects and fine-grained recognition.
In recent years, vision transformers have demonstrated remarkable superiority to CNNs in single image super-resolution (SISR), yet their heavy computational and storage costs hinder practical deployment. In this work, we revisit CNN-based residual learning for SISR and identify a specific yet overlooked problem: the residual signal is often biased by mapping errors during model training, and this bias can propagate and accumulate through layers. To this end, we revisit the commonly used residual learning and feature fusion in SISR and propose an adaptive residual correction network (AdaRCN) in this work. First, we introduce a residual correction mechanism that adaptively compensates for the bias in the residual, which is utilized to ease error accumulation and improve mapping accuracy. On the other hand, we generalize the standard identity shortcut to a weighted channel concatenation followed by a 1 × 1 convolution, which is a more versatile strategy for adaptive feature fusion. Our AdaRCN is built entirely upon a naive CNN without complex architecture design and training strategies, thus ensuring efficient inference and parallelization. Extensive experiments verify the benefits and effectiveness of residual correction and adaptive feature fusion in improving the representational capability of our model, enabling it to achieve impressive performance comparable to advanced SISR models with moderate overhead.
Jinsha Tian, Junyu Guo, Bing-Bin Feng et al.· Italian National Conference...· 0 citations
Image super-resolution aims to reconstruct high-resolution images from low-resolution inputs and is a core reconstruction task in computer vision. Traditional discrete residual network-based methods realize feature transformation by stacking a fixed number of layers, but are limited by the discrete depth, making it difficult to capture continuous image degradation and reconstruction dynamics. In recent years, Neural Ordinary Differential Equations (ODEs) have received widespread attention for their continuous depth modeling capability, but face challenges such as high computational cost and large feature dimensions in super-resolution tasks. In this paper, we propose Adaptive ODE-ResNet (Adaptive ODE-ResNet), which reconstructs the residual block into a continuous ODE-driven process to realize flexible and accurate feature evolution. The core innovations include two points: first, the space complexity-aware solver (SCAS), which dynamically adjusts the ODE integration step according to the local texture complexity, reducing the computational overhead in flat regions and improving the reconstruction accuracy in texture-rich regions, and bringing a + 0.55 dB gain; and second, the low-rank ODE projection (LROP), which compresses the high-dimensional ODE features through low-rank decomposition, with an accuracy loss of less than 0.05 dB under fourfold compression. is lower than 0.05 dB. In addition, an independent ODE function design enhanced by CBAM channel-space dual-attention mechanism is proposed to realize fine feature evolution with 640-channel feature space together with 24-step RK4 integration. The method achieves 29.27 dB PSNR and 0.8107 SSIM on the DIV2K ×4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times 4$$\end{document} benchmark; on the × 2 task, it achieves 38.30 dB PSNR on Set5, which exceeds the ODE-SR method (38.21 dB) and approaches the EDSR level, verifying the validity and competitiveness of the adaptive ODE framework with multiple magnification factors. The framework provides an accurate, efficient and scalable continuous-time modeling scheme for high-resolution image reconstruction.
Hai-Ying Zhang, Liangping Tu· Journal of King Saud Univers...· 0 citations
Residual Flow Matching for Image Super-Resolution (RFMSR) is proposed, a vision-only framework that centers the source distribution at the LQ latent, reducing transport distance and preserving structural priors throughout the flow trajectory.
Shuwei Huang, Tianyao Luo, Jicheng Liu et al.· arXiv.org· 1 citation
Fine-grained visual understanding depends on local detail, yet visual encoders face a trade-off between costly full-image high-resolution processing and compact global encoding that can weaken such evidence. Inspired by human active vision, we separate where to look from what to encode. We focus on the latter and introduce FAVE (Foveated Adaptive Visual Encoding), a lightweight variable-resolution ViT that encodes externally selected regions at high acuity while preserving native geometry. We first isolate this encoding problem using oracle ground-truth crops in a controlled small-object regime. On ImageNet objects with a native maximum side of 96 pixels, FAVE improves Top-1 by 9.4 points over a fixed-resolution ViT on the same crop window with 12.7 times lower FLOPs. Increasing global resolution or backbone capacity does not recover the same operating point. We then integrate FAVE as a complementary local branch in FastVLM. Its local tokens are combined with FastVLM's global visual tokens, while the original global pathway and language model remain frozen. With at most 16 additional local tokens, FAVE improves TextVQA by 1.60 points and achieves a 3.3 times controlled TTFT speedup over SmolVLM2-2.2B. On GQA attribute questions, it improves FastVLM-1.5B by 1.31 points, extending the benefit beyond text while narrowing the gap to FastVLM-7B. Together, these results show that selectively allocating high-acuity local capacity provides an efficient complement to broader global representations and model scaling for fine-grained understanding of small objects, text, and attributes.
High-resolution video diffusion models built on Diffusion Transformers (DiTs) deliver strong fidelity but quickly exhaust the memory budget of a single workstation. A 100 billion-plus parameter DiT easily requires over a terabyte of persistent state, while naive spatiotemporal self-attention grows quadratically in sequence length. These two walls -- parameter memory and activation memory -- prevent researchers from adapting massive generative models without large GPU clusters. We revisit this problem from a systems perspective and introduce MegaSlide-DiT, a prototype that demonstrates how a pre-trained 105B DiT can be adapted on a single H200 GPU with 1.5 TB of host RAM. Our key insight is that the GPU need not own the model state: all persistent weights, master weights and optimizer moments remain in host memory, while only transient shards are streamed to the GPU on demand. Simultaneously, we replace quadratic global attention with 3D Deformable Slide Attention (3D-DSA), a motion-adaptive local attention operator that reduces both memory and computational complexity to linear in the sequence length. We report detailed memory accounting, execution traces and evaluation results to substantiate our design. MegaSlide-DiT does not claim to train a 105B model from scratch on a single GPU, nor does it magically solve bandwidth limits; rather, it offers a pragmatic path for full-parameter adaptation of massive video diffusion models on high-end workstations.
Vision Mamba models replace quadratic self-attention with linear complexity selective state space models (SSMs), emerging as efficient visual backbones. However, MambaOut demonstrates that a Gated CNN block can match or exceed VMamba on image classification, questioning the necessity of SSMs for vision. This raises a fundamental question: do VMamba and MambaOut encode visual information differently at the representation level? To investigate, we apply cross model centered kernel alignment (CKA) analysis and find that VMamba's final stage blocks form representations distinctly different from both MambaOut and its own preceding blocks. We therefore focus on the final block features, decomposing each spatial token into magnitude and direction. MambaOut concentrates class-discriminative information in high-norm foreground tokens that align with Grad-CAM attribution. VMamba, by contrast, produces high-norm tokens predominantly in background regions, misaligned with Grad-CAM, yet preserves discriminative signals primarily in token directions. These observations reveal that the two models rely on different encoding strategies. We connect this difference to high-resolution classification and semantic segmentation. VMamba distributes logit support broadly across object regions, whereas MambaOut relies on sparse dominant tokens, a strategy that becomes less stable as token counts grow. Under full fine-tuning for segmentation, VMamba consistently outperforms MambaOut. These results suggest that VMamba's advantage in dense prediction stems not merely from the SSM mechanism or sequence length, but from how semantic evidence is organized across token magnitude, direction. Ultimately, we conclude that token magnitude and directional structure serve as critical axes for improving visual backbones, particularly under dense supervision.
Jinyan Yu, Juyoun Park· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
