Danilo P. Mandic

Surviving Resource-constraint Compression: Capability Retention under Tensor-train Decomposition for Sub-billion Parameter Language Models

This work proposes a training-free model compression approach based on the tensor-train decomposition, whereby each pre-trained token embedding is converted into a lower-dimensional matrix product state (MPS), and comprehensively investigates what language capabilities are preserved under training-free compression at different compression ratios.