A two-stage framework is developed, in which an approximately orthonormal neural basis is first constructed, and the output weights are subsequently determined using least-squares methods.
Abstract
We propose a Sparsely Connected Rank-Inspired Neural Network (SCRINN) for function approximation and the numerical solution of partial differential equations (PDEs). By introducing a structured sparse connectivity architecture and an orthogonality- and normalization-guided loss function, SCRINN constructs an approximately orthonormal neural basis while promoting progressive rank growth of its evaluation matrix and reducing the number of trainable parameters. A two-stage framework is developed, in which an approximately orthonormal neural basis is first constructed, and the output weights are subsequently determined using least-squares methods. Numerical experiments on function approximation and a variety of steady-state, time-dependent, and nonlinear PDEs demonstrate that SCRINN consistently achieves high approximation accuracy with substantially lower training costs. The code is publicly available on GitHub.
Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keeping the matrices dense, and thus efficient on standard hardware. Existing methods, however, choose the subspace to remove from each weight matrix with local closed-form crit...
Massimo Bini, Anders Christensen, S. Alaniz et al.· 0 citations
We study spectral graph neural networks built from Hermite polynomials and propose HermNet, a simple model that combines a nodewise predictor with normalized Hermite propagation. Its sparse recurrence requires neither eigendecomposition nor a learned basis. We distinguish the basic model from optional coordinate calibr...
Implicit neural representations (INRs) are shaped by the spectral structure induced by their input encodings and activation functions. Existing methods improve fitting primarily by modifying which frequencies are available to the network, through coordinate encodings or periodic nonlinearities. However, frequency acces...
Tamir Shor, O. Litany, Alex M. Bronstein· 0 citations
This paper proposes a novel Tensor Train (TT)-based tensor-on-tensor regression optimization framework for variable selection based on mode-1 hyperslice sparsity, and designs an alternating iterative algorithm equipped with a preconditioned metric to efficiently solve the proposed model.
This paper analyzes the already established Linearly Scaled Hyperbolic Tangent function and proposes ReXTanh, a smooth-sparse hybrid function that provides C 1 continuity while remaining asymmetric, showing that keeping negative magnitudes can outmatch forced sparsity in harder recognition problems.
Emil Barciś, K. Rojek· Journal of Applied Mathemati...· 0 citations
It is proved that the constrained decoupled objective is equivalent at optimality to a specific joint penalty on the inner and outer weights, for any positively homogeneous activation, and thus admits a clean projected and proximal formulation.
Charles Kulick, Armenak Petrosyan, Tang Sui· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
