An Efficient Fully Hardware DQN Accelerator With Parallel Q and Target Networks for Low-Latency Edge AI Application

Abstract

Deep reinforcement learning (DRL) workloads such as Deep Q-Networks (DQNs) require repetitive forward evaluations of both the Q-network and the target network, which leads to high training latency and inefficient resource utilization on edge System-on-Chip (SoC) platforms. This paper presents an efficient parallel double network DQN accelerator that performs a fully parallel forward computation of the Q-network and the target network using shared processing elements (PEs) for time-multiplexed neuron computations in each network. The proposed architecture employs fixed-point quantization, a parallel neural network engine, and on-chip BRAM-based parameter and experience replay memories to eliminate forward-path serialization and off-chip memory bottlenecks. A lightweight single-cycle XORShift pseudo-random number generator is integrated to support stochastic policy exploration and randomized replay addressing without introducing timing overhead. All system parameters, including operation modes, network dimensions, and memory depth, are configured through AXI-accessible registers by the processing system. Implemented on an Ultra96-V2 FPGA platform, The proposed accelerator achieves a training throughput of 231 KIPS, corresponding to a $1.3\times $ improvement over the serialized DQN design, with only an 8% reduction in area efficiency (KIPS/1KLUT). By executing all DQN computations entirely in hardware and supporting configurable inference and training modes, the proposed architecture effectively reduces software intervention and provides a flexible solution for accelerating DQN workloads on FPGA platforms. Experimental results demonstrate that the proposed design preserves learning convergence while reducing training latency, making it well suited for low-power edge DRL applications for autonomous agents.