This work defines first-order residual depth saturation as the absence of a strict local decrease from every admissible insertion, and proves residual non-degeneracy is necessary and sufficient: additional depth has first-order value exactly when conditional activation gradients have a nonzero projection onto at least one admissible residual tangent space.
Abstract
How can we determine whether a trained neural network is already deep enough? We study this under a fixed function-preserving residual-growth protocol specifying insertion locations, residual families, zero-output initializations, and zero-state first-order updates. We define first-order residual depth saturation as the absence of a strict local decrease from every admissible insertion. We prove residual non-degeneracy is necessary and sufficient: additional depth has first-order value exactly when conditional activation gradients have a nonzero projection onto at least one admissible residual tangent space. This boundary is shared by descent-compatible zero-state updates and invariant under regular local reparameterizations preserving that tangent space. Under residual-signal realizability, raw activation-gradient vanishing exactly certifies saturation. Across ResNets, GPT-2-style models, and continued-pretrained Pythia checkpoints, the maximum activation-gradient norm decreases toward a low-signal regime with depth. Function-preserving growth also achieves converged performance competitive with training from scratch. These results support activation-gradient magnitude as a conservative diagnostic of the remaining empirical first-order value of residual depth.
It is shown that adding a free affine skip connection leaves these function-space solutions unchanged but fundamentally improves the parameter-space landscape: every KKT point of the constrained problem becomes globally optimal, whereas suboptimal KKT points can occur without the skip connection.
Karolina Drabik, Ben Lewis, Antoni Puch et al.· 0 citations
Parameter-efficient fine-tuning (PEFT) determines not only how many parameters are trained, but also which local directions a model can move in, so similar adapters can affect subgroup losses differently. Since curvature matrices are infeasible to form at adapter scale, scalar summaries such as the Fisher trace are oft...
Wasif Jalal, Sachin Deb, Asif Salekin· 0 citations
How many directions does a neural representation use to encode a concept? A common answer repeatedly erases probe directions and reports the stopping count or cumulative removed rank. We show that both quantities can change under an information-preserving invertible reparameterization, so neither is intrinsically a con...
Tingan Jin, Shu-Hang Dong, Hao-Song Li et al.· 1 citation
Recursive computation repeatedly compresses or reuses intermediate states, creating a simple tension: information that must remain useful across longer recursive paths is also exposed to more opportunities for loss before reaching the final prediction. Existing reconstruction or local-prediction objectives provide trac...
Pei-Lin Wang, Shi-Yang Feng, Hong-Fu Gao et al.· 0 citations
Large sparse linear systems from PDE discretizations require correction subspaces whose operator images explain the current residual. We study this residual-image viewpoint and propose Gate-RINS, a neural subspace solver that generates polynomial correction bases from cached residual probes and modulates them with a li...
Zhong-Yan Ouyang, Wei-Xin Liao, Ming-Quan Feng et al.· 0 citations
Results indicate that NObSP complements scalar attribution methods by recovering functional contribution profiles with separable positive and negative evidence.
A. Caicedo, Victor de la Hoz, Santiago Alférez· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.