Asynchronous Co-Execution of PyTorch on Zynq-7000: FPGA Matrix Delegation and PS–PL Overlap for End-to-End Inference Throughput
TL;DR
A systolic array-based accelerator prototype implemented on the Zynq-7000 SoC integrated directly into PyTorch, enabling dense linear algebra to be delegated to the FPGA chip while Cortex-A9 continues executing the software stack uninterrupted.
Abstract
Embedded systems increasingly require on-device deep learning, yet their processors must simultaneously handle real-time sensing, networking administration, and data control. Existing Field-Programmable Gate Array (FPGA) accelerators typically target peak per-operator throughput without addressing concurrent execution demands of real-time embedded platforms. This paper presents a systolic array-based accelerator prototype implemented on the Zynq-7000 SoC integrated directly into PyTorch, enabling dense linear algebra to be delegated to the FPGA chip while Cortex-A9 continues executing the software stack uninterrupted. Unlike traditional accelerators optimized for peak per-operator speed, this design prioritizes asynchronous co-executionbetween the processing system (PS, the dual-core Cortex-A9) and the programmable logic (PL): The PL performs tiled matrix multiplication, while the PS executes preprocessing, orchestration, and I/O data concurrently, increasing effective end-to-end throughput regardless of the relative isolated performance of CPU and FPGA. The proposed module includes high-level-synthesis (HLS)-based matrix multiplication, activation functions, and Advanced eXtensible Interface (AXI)-Stream Direct Memory Access (DMA) interfaces, wrapped as custom PyTorch kernels under the PetaLinux operating system. The results obtained on the PYNQ-Z2 board show that, once the DMA transfer time is included in the measurement, the FPGA path does not surpass Cortex-A9 in isolated per-operator latencies across the evaluated range; the benefit lies instead in delegating the matrix compute to the fabric at low incremental power while the host CPU cores stay available for concurrent tasks. A concurrent workload sweep across matrix sizes from 8×8 to 256×256 confirms that the co-execution mode sustains 98–99% of available PS compute throughput compared with a constant ≈50% in single-core blocking mode; the difference is statistically significant for all evaluated sizes (see Mann–Whitney U: U=25, p=3.97×10−3, perfect discrimination, n=5). A fair dual-core CPU-only baseline attains comparable PS availability, so this figure reflects the dual-core scheduling that co-execution enables rather than a per-operator advantage of the fabric; the accelerator’s distinct role is to perform the matrix arithmetic off the general-purpose cores at low incremental power. The design occupies only 8% of available look-up tables (LUTs) and 5% of digital signal processing (DSP) blocks, maintains 1.69 W power with a junction temperature of 44.5 °C, and achieves 96.10% MNIST accuracy under fixed-point arithmetic.