Binary Representation of NLP Embeddings Using Spiking Neural Networks*

Abstract

Efficient text embedding is critical for large-scale natural language processing (NLP) applications, where storage, computation, and scalability are key concerns. While binary representations offer a compact alternative to real-valued embeddings, existing approaches typically rely on fixed or heuristic thresholding schemes that fail to capture task-relevant structure in the data. In this paper, we propose a spiking neural network (SNN) based framework for learning binary representations of pretrained NLP embeddings, where binary spiking codes are learned directly from continuous transformer embeddings using surrogate gradient optimization. Unlike conventional threshold based binarization methods such as min-max scaling, Otsu thresholding, and hybrid heuristics, the proposed approach predicts the projection parameters and these are optimized jointly with the downstream task, enabling adaptive and task-aware binarization. The binary embedding behaves like a text DNA. We evaluate our method on a wide range of benchmark NLP datasets, including IMDb, AG News, SNLI, and additional text classification tasks, and compare it against multiple heuristic binarization techniques as well as real-valued embeddings. Extensive experimental results demonstrate that the proposed spiking binary representations consistently achieve superior classification performance, outperforming existing thresholding methods and, in several cases, exceeding the performance of real-valued embeddings.