High-Utilization, High-Flexibility Depth-First CNN Coprocessor for Image Pixel Processing on FPGA

Recently, CNNs are increasingly exploited for pixel processing tasks, such as denoising, which opens up new challenges due to the increased activation and operation count. This article presents a CNN coprocessor architecture to solve these challenges on field-programmable gate array (FPGA) through four main contributions. First, the I/O communication between the host processor and the FPGA is reduced to a minimum using a depth-first (DF) principle. Three new DF approaches are presented. Second, to ensure high throughput, the increased parallelization opportunities of the proposed line-based DF operation are analyzed. Third, introducing programmability to the compute array is introduced to enable a broad deployment while maintaining high utilization of the available multipliers digital signal processings (DSPs), independently of the kernel dimensions and without control of the host processor. This is in contrast with many state-of-the-art FPGA implementations, focusing on only one algorithm and/or one kernel topology. Fourth, a model is built to investigate the influence of architecture parameters and show the benefits of DF. The scalable design can be deployed on a wide range of FPGAs, maintaining 78%–93% DSP utilization across all algorithms (denoising, optical flow, depth estimation, segmentation, and super-resolution) and FPGA platforms. Up to 695 GOPS is achieved on a Zynq XCZU9EG board, matching state-of-the-art performance with a more flexible design. The throughput is compared with other pixel processing architectures on FPGA.