A Scalable, Ultra-Low-Latency FPGA Accelerator for Neuromorphic Feature Extraction

Abstract

Neuromorphic algorithms can extract features directly from an asynchronous event-based stream by exploiting complex spatio-temporal relationships without frame-reconstruction overhead. These event-based streams are created by neuromorphic sensors that capture events at a microsecond resolution with variable bandwidth. Therefore, these algorithms require low-latency processing architectures, like field-programmable gate arrays (FPGAs), to enable real-time machine-learning applications. A Hierarchy Of event-based Time Surfaces (HOTS) is one such algorithm that exploits spatio-temporal relationships between events using decay functions and <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-means clustering. The sequentially dependent nature of temporal processing presents significant challenges when designing an efficient stream-processing accelerator. Thus, less-complex spatial algorithms, like <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-means clustering, can be applied to event-based streams to enable lower-latency feature extraction with neuromorphic sensors. This research showcases the performance of the HOTS and <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-means clustering algorithms on feature-extraction tasks using neuromorphic-sensor data with FPGAs using state-of-the-art high-level synthesis tools. Additionally, a dataset is developed and considered to evaluate practical sensor event-generation rates. While using time-restricted sampling, spatial <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-means clustering outperforms the spatio-temporal HOTS by achieving competitive accuracies of 91.8%, 63.4%, and 99.9% on N-MNIST, IBM DVS128 Gesture, and custom datasets. Using an Altera Stratix 10 FPGA, our accelerator showcases an event-processing latecy of 65 ns and a throughput of 15.38 Mevt/s, processing events over <inline-formula> <tex-math notation="LaTeX">$2.2\times $ </tex-math></inline-formula> faster than the practical sensor event-generation rate. By using less than 3.0% of FPGA-board resources, our spatial <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-means clustering FPGA design proves to be a scalable, ultra-low-latency accelerator for neuromorphic feature extraction.