Energy-efficient FPGA acceleration of reinforcement learning algorithms using high-level synthesis
Low latency and energy-efficient design are important requirements for Reinforcement Learning (RL) algorithms deployed on embedded and edge computing systems. Software based Q-learning shows increased power consumption as well as the overhead of sequential execution. The paper shows an energy-efficient FPGA architecture for the Q-learning update based on High Level Synthesis (HLS). An FPGA accelerator targeting the device platform Xilinx Zynq-7000 (xc7z020clg400-1) has been designed and implemented. Loop pipelining and precision fine-tuning design techniques were applied to optimize the architecture. This improvement was achieved by 50 percent as a baseline architecture got a latency of 14 cycles, which was reduced to 7 cycles when the architecture was pipelined with loops. A relative comparison of fixed point (ap_fixed<16,8>) and floating point implementations has shown that the floating point arithmetic increases the number of LUTs used by 4 times, the number of DSPs used by more than 2 times, and the delay by 29 cycles. Post implementation power analysis for fixed-point architecture shows that its energy consumption is about 4 times less per Q-update operation than a floating-point design. The results show that pipelined fixed-point FPGA architectures are a good solution to speed up the reinforcement learning process in resource-limited embedded systems.