Jul 2026· Annual International Computer Software and Applications Conference· pp. 1453-1462· 0 citations· 27 references
Abstract
Traditional model reduction techniques for neural network compression usually leverage existing matrix/tensor decomposition methods to explore the redundancy among model parameters, which may overlook the local structures of the models. In this work, we propose a novel low-rank approximation algorithm, called Dimension-Augmented Tensor-Train (DATT) decomposition, which integrates dimension slicing with the Tensor-Train decomposition. DATT allows different partitions to be represented using distinct, non-shared spanning sets. This enables the decomposition to adapt more flexibly to localized structure, potentially achieving better compression. With the localized low-rank structures, DATT provides a more compact representation of neural network weights. We evaluated the performance of DATT with the other state-of-the-art low-rank approximation-based model compression algorithms, and demonstrated that DATT outperforms them in terms of parameter reduction and model accuracy. In particular, for compressing ResNet-20 on CIFAR-10, DATT achieves a 77.00% reduction in parameters with only a 0.37% drop in top-1 accuracy.
This work presents a method to accelerate the optimization of learning high dimensional functions using deep neural network (DNN), and studies the effect of adding features which distill pretrained DNN into TNs using a discretize and decompose strategy.
This work proposes a training-free model compression approach based on the tensor-train decomposition, whereby each pre-trained token embedding is converted into a lower-dimensional matrix product state (MPS), and comprehensively investigates what language capabilities are preserved under training-free compression at d...
Mingxue Xu, Y. Xu, Danilo P. Mandic· Machine Intelligence Researc...· 0 citations
SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm, is introduced.
David González-Martínez, Shiwei Liu· arXiv.org· 0 citations
This paper proposes a novel Tensor Train (TT)-based tensor-on-tensor regression optimization framework for variable selection based on mode-1 hyperslice sparsity, and designs an alternating iterative algorithm equipped with a preconditioned metric to efficiently solve the proposed model.
This work presents the tree tensor network Nyström (TTNN), an algorithm that extends recent research on streamable tensor approximation to the more general tree tensor network format, enabling a unified treatment of various existing methods.
Alberto Bucci, Gianfranco Verzella· Numerical Linear Algebra wit...· 3 citations
This work introduces a novel and efficient solver for the underlying HJB equation based on the functional tensor train (FTT) format, which leverages latent low-rank structures to efficiently approximate high-dimensional functions, enabling both model compression and rapid computation.
Robert Gruhlke, Julius Berner, David Sommer et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
