Deadline-Guarded Edge Inference With FPGA Preemption

Abstract

Deep neural networks (DNNs) are increasingly being employed in delay-sensitive edge applications such as autonomous driving, industrial automation, and extended reality. However, due to the use of complex computing hardware and algorithms, the execution time of a DNN is stochastic and follows a distribution usually with a long tail exceeding the specified deadline. To guard the deadline, we propose a real-time and preemptive GPU-FPGA heterogeneous computing system that continuously monitors the progress of the DNN execution on the GPU and predicts deadline miss. Upon prediction of a miss, the system activates the FPGA as a deadline guardian to run a smaller DNN for the same task to meet the deadline. We design lightweight predictors based on offline data of progress versus final completion time and hardware status readings. Moreover, to deal with the performance loss due to resource contention on the FPGA, we further propose a simulation method to find the optimal preemption plan in multitasking scenario. Extensive evaluation with a Jetson AGX Orin GPU and a Xilinx ZCU102 FPGA shows that our system achieves up to 20 times reduction on miss rate and improves effective accuracy by more than 10%.