Geometric Supercharger: Data-Informed Hyperbolic Initialization of MLP-based Sparse Neural Networks

Abstract

Sparse neural networks can substantially reduce computational and memory costs, yet their performance remains highly sensitive to the choice of sparse topology, particularly at extreme sparsity, where they struggle to outperform fully connected models. Existing sparse initialization methods are predominantly structure-driven and rarely conditioned on the data, motivating a central question: how can sparse connectivity be organized in a principled way that reflects the structure of the data rather than relying on predefined or heuristic topologies? Network science has shown that complex network topology can be characterized by an underlying latent geometry, helping to explain processes such as network routing, navigation, and dismantling. This suggests that, rather than imposing a predefined sparse topology, one can extract the latent geometry of the data and use it to organize network connectivity. Motivated by this connection, we introduce Geometric Supercharger (GS), a data-informed framework for hyperbolic initialization of MLP-based sparse neural networks. GS extracts pairwise feature relationships using Pearson correlation and constructs a macro-scale geometric embedding through shortest-path-based pre-weighting and dimensionality reduction. The resulting geometry is then reused to reweight the network and refine the embedding, thereby supercharging it. This supercharged embedding is followed by meso-scale refinement based on community detection and angular reordering. The final angular coordinates encode data-informed node similarity and are used to generate a sparse hyperbolic topology. Building on previous findings that arbitrary, data-agnostic hyperbolic hierarchies can hinder sparse neural network training, we show that making the hierarchy data-informed turns the same geometric prior into a performance advantage. We evaluate Geometric Supercharger across four diverse tasks: image classification, physics-informed neural networks, reinforcement learning, and stock-market prediction. GS achieves state-of-the-art performance among all evaluated static sparse initialization methods in the ultra-sparse 99% regime and outperforms fully connected networks in the majority of experiments at 90% and 99% sparsity. These findings establish data-informed latent geometry as an important organizing principle for sparse MLP initialization and motivate a broader class of geometry-driven sparse neural networks.