This work introduces a method that induces sparse neural activity in heavily quantized linear-attention models with minimal performance loss, and positions sparse, quantized linear-attention models as a natural fit for deploying LLMs on event-driven multi-core platforms.
Simon Richter, Ruhai Lin, Jason Yik et al.· 0 citations
This work proposes heterogeneous neural networks that combine spiking neural networks (SNNs) and artificial neural networks (ANNs) at bandwidth-limited regions, such as chip boundaries, where spike-based communication reduces data transfer overhead.
Joshua Nardone, Rui-Jie Zhu, Ruhai Lin et al.· International Conference on...· 0 citations
Results show that the benefits of recurrent depth can be retained without repeatedly executing the dense FFN, and view recurrence as repeated composition of a state update and argue that an application is valuable when it exposes a new cross-position influence direction that remains observable at the task readout.
Ruhai Lin, Yi-Yang Guo, Ruiming Zhu et al.· 4 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.