Hi I am new to the world of convolutional neural networks and would like to implement a 2D convolution operation using the sliding window approach on a xilinx FPGA. The input to the image is a 32x32 image with 2 channel on which 6 kernels of 5x5 is convolved to produce 6 output feature maps. Now assuiming I have sufficient DSPs on the FPGA, how would I parallelise the problem? After doing some research I have understood that we can either parallelise over the input or output feature maps or the kernel or both. For a 5x5 kernel, I would need 25 multiplications and 25 additions including the bias. If I have 25 DSPs operating in parallel, I can achieve this in one clock cycle. Is my understanding of the problem of parallelization correct up to now?

Now considering the input is stored in buffers and streamed to my convolution module and weights are pre loaded to the module, how is the sliding window computation performed? I realise I would have to use counters to keep track of the position of the input till the end of the N_W and N_H respectively. There is quite a lot of literature about implementing this using systolic arrays of multipliers but I am not sure I get those.

Could someone help me understand a dataflow for the convolution operation?

Any help would be greatly appreciated. An eager student.

  • 1
    \$\begingroup\$ nothing in your problem statement suggests that using an FPGA here is a good idea. Choosing the tool for a job first before defining requirements is a typical beginner's mistake. Can you explain how you came to the conclusion that designing an FPGA image to do that was wiser than e.g. to write CPU or GPU software, which is much easier, and in case of such miniature data sizes, probably not even worse, power-wise? \$\endgroup\$ – Marcus Müller Oct 15 '18 at 16:07
  • 1
    \$\begingroup\$ This is to process upto a 1000 images per second and preferably in future to enable a mobile inference device development... \$\endgroup\$ – frisco_1989 Oct 15 '18 at 16:13
  • 1
    \$\begingroup\$ I think it would be more efficient to perform 2D FFT on the grayscale image and once on the kernel and simply multiply them together, element by element, in frequency domain, and then perform an inverse 2D FFT. - Remember, multiplication in time domain is convolution in frequency domain. Multiplication in frequency domain is convolution in time domain <-- do dis. \$\endgroup\$ – Harry Svensson Oct 15 '18 at 16:13
  • 1
    \$\begingroup\$ I think the question is not very clear because I forgot to mention that this is an attempt to do a forward path inference of a deep neural network... specifically a convolutional neural network... Sorry for the misunderstandng \$\endgroup\$ – frisco_1989 Oct 15 '18 at 16:16
  • 3
    \$\begingroup\$ @HarrySvensson: I think you can read that as "FPGA now, maybe ASIC later". \$\endgroup\$ – Dave Tweed Oct 15 '18 at 16:38

The problem of processing a 2-d kernel of data over a large dataset (not just convolution) comes up so regularly in HD video processing that I came up with a generic way of handling it that I use all the time.

I developed a generic "kernel generator" that uses line buffers and registers to present all of the input data for a given output pixel in parallel. An N×N kernel requires N-1 line buffers and N-1 registers. It assumes that the data is arriving in "raster-scan" order, like a TV signal.


simulate this circuit – Schematic created using CircuitLab

The next stage could be your multipliers, but more often than not, I need to handle the edges in some special way, such as zeroing out the values that fall outside the input data, or reflecting the data across the edge, or whatever. Therefore, I have some standard modules (N2 pixels in, N2 pixels out) that consist of counters and multiplexers that do this edge processing before passing the data to the actual data processing module.

For a convolution, you can do all of the multiplies in parallel in the same clock period, but then the adds will have to be pipelined. For example, if you can only add two numbers in one clock period, you'll need a "tree" of adders that's 5 levels deep for a 5×5 kernel. If you can add three or four numbers at a time, you will only need 3 levels of pipeline.

Obviously, the same kernel generator can feed multiple convolutions in parallel if that's what you're doing, but I second Harry Svensson's notion of using FFT techniques if you're doing more than a few of them.

  • \$\begingroup\$ Thank you for the answer. If you dont mind me asking some really basic questions.... when you say buffer, you mean a FIFO? and U2,3,4,5 are registers? I am sorry but I dont understand how the data is presented parallely to the next stage of multipliers here? \$\endgroup\$ – frisco_1989 Oct 16 '18 at 14:27
  • \$\begingroup\$ Yes, the "line buffer" is a large (wide) FIFO. If you "unwrap" the data path, you'll find that it's really just a really long delay line that has all of the correct taps to give you the input kernel data for any given output position. The reason that the registers U2 through U5 exist is that the line buffer is implemented using RAM, which doesn't have enough ports to access all of the taps simultaneously, so the extra registers create extra copies of the data that can be accessed in parallel. \$\endgroup\$ – Dave Tweed Oct 16 '18 at 15:02
  • \$\begingroup\$ Ok.. Now i get it. But then now I have to wait 25 clock cycles to get my first output and then I get the mutipliers output every clock cycle? \$\endgroup\$ – frisco_1989 Oct 16 '18 at 15:53
  • \$\begingroup\$ You actually have to wait 2 whole lines plus 2 pixels before you get your first valid output, but yes, after that, you get one result every clock. \$\endgroup\$ – Dave Tweed Oct 16 '18 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.