Zheng-Zheng Tang

NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling

NeuronSpark is a self-attention-free language model built around input-dependent spiking membrane dynamics, where each neuron's decay rate, gain, and firing threshold adapt to the current token content, which enables billion-scale pretraining from random initialization without backbone transfer, conversion, conversion,...