A cross-modal alignment IP for accelerating open-vocabulary object detection inference on heterogeneous edge platforms

Abstract

The cross-modal similarity calculation between large-scale visual features and text dictionaries in the Open-Vocabulary Object Detection (OVOD) inference stage exerts a significant degree of memory access pressure and computational overhead, which consequently becomes a primary bottleneck that limits the real-time performance of the system. To address this problem, this paper presents the design of a dedicated acceleration Cross-Modal Alignment IP accelerator for the CPU+NPU+FPGA heterogeneous edge platform. This is used to accelerate the cross-modal alignment calculation in the OVOD inference stage. The proposed method involves the transformation of the text feature generation from online processing to offline processing, and the combination of the on-chip BRAM dictionary resident mechanism and streaming data supply method. The result of implementing these changes is a significant reduction in repeated access to external storage in the inference stage. Concurrently, the execution efficiency of the similarity calculation between large-scale visual features and text semantic vectors is enhanced by the parallel dot product reduction array and write-back optimization design. The experimental findings, derived from the utilization of the FMQL30TAI heterogeneous platform, demonstrate that the proposed scheme attains a Pearson correlation coefficient of 0.998 with the software floating-point benchmark. This is accomplished while preserving substantial numerical consistency. The proposed scheme accomplishes approximately 41 times faster in the alignment stage and reduces the end-to-end single-frame processing delay of the system from 220 milliseconds to 41 milliseconds. The overall throughput has been increased to 24.4 FPS. The findings demonstrate that the proposed method can effectively alleviate the memory access and computing bottlenecks of large vocabulary OVOD in edge deployment, and provide a feasible hardware acceleration scheme for the application of open vocabulary detection in real-time embedded vision scenarios.