Lightweight Redesign of Long-Used Operators in Vision Backbones for Efficient Visual Recognition

Abstract

Recent vision backbones increasingly rely on sophisticated modules, whereas long-used operators such as residual connections, activations, and normalization layers remain less explored for lightweight redesign. This paper revisits these operators and proposes three operator-level redesigns: Subtractive Residual Connection (SRC), Learnable Gating Response Function (LGRF), and statistics-discrepancy-guided Dynamic Dual Normalization (DDN). SRC changes shallow residual fusion from addition to subtraction to suppress redundant responses and induce attention-like response focusing without an explicit attention branch; LGRF extends fixed gating activations to channel-wise learnable response curves; and DDN generates sample-level LayerNorm (LN)–BatchNorm (BN) fusion weights from input statistics and LN-BN discrepancy. On ImageNet-1K, SRC improves ResNet models without extra parameters or floating-point operations (FLOPs). Applying all three proposed methods to MambaOut-Femto improves Top-1 accuracy by 1.01 percentage points with only 0.06 M additional parameters and 0.01 GFLOPs. Ablations on ImageNet-100 and CIFAR-10/100 support effectiveness and stability, while DeepWeeds validation further supports the practical value of all three operators for weed recognition under complex natural backgrounds. Gradient-weighted Class Activation Mapping (Grad-CAM), feature response maps, learned response curves, and dynamic LN-BN weights support module interpretation. Overall, revisiting long-used fundamental operators remains valuable for efficient visual recognition.