# FPGA: efficient way to collect data from many cores within SoC

I am looking for an efficient (in terms of footprint) way to periodically collect data from a multitude of processing elements (PEs) within an Spartan6-based design.

I currently have 16 PEs, but when the design scales up it may become up to 64. Each PE outputs two 16-bit words of data (A and B) at a frequency of ~100 KHz. The output data from all PEs needs to be collected and added to two sums (A_sum and B_sum). The system clock will be >100 MHz, so I have >1000 clock cycles to collect the data from the PEs.

The easiest way to do it would be two 16-bit wide multiplexers (A_mux and B_mux) with one input per PE each. For 16 PEs, this would cost 128 slices plus 512 interconnect lines within the chip, and it would take 16 clock cycles to collect the data. Since it doesn't need to be that fast, I am wondering if there is a more resource-efficient way to do it.

I was thinking of serializing the data in some way. This would save interconnect lines, but I would use more slices to implement the serializers in every PE. I would also need shift registers with parallel outputs (hence no SRL32) for de-serialization.

UPDATE

Some of you suggested I may have answered my own question. Serialization obviously comes to mind, however I am not sure whether it is really going to be more efficient. My question is not so much whether serialization is feasible, but I am concerned that serializers for every PE will use up more FPGA resources than a 16-bit multiplexer with inputs for every PE. Maybe there is some clever implementation I haven't thought about, but it seems to me that the fat mux is really the most resource-efficient way. Am I wrong?

• "I am looking for an efficient (in terms of footprint)" can be replaced by this text: "I am looking for an efficient (in terms of dinosaurs)". – Harry Svensson Jan 24 '18 at 7:41
• Yes, serialising sounds good. Looks like you've answered your own question. You can keep the data as serial as you add it, serial-bit-wise, into serial accumulators. You do not need to go parallel until your final readout. Any hardware designer uses the full spectrum from fast, wide parallel operations to slow, narrow, footprint efficient serial operations, to optimise the speed/footprint tradeoff as required. You only use parallel operations at slow clock rates when you have the luxury of an early design in an under-filled FPGA and you want to get it working yesterday with no brain-strain. – Neil_UK Jan 24 '18 at 7:47
• @HarrySvensson, code footprint (bytes used), PCB footprint (area on board), FPGA footprint (gates used)... Seemed kinda obvious to me, anyway :-) – TonyM Jan 24 '18 at 8:28
• @TonyM Okay, so now there's even more parameters that it could be that I didn't think of. Which one is "in terms of footprint"? - Doesn't really matter since everybody seems to understand everything anyways ;). – Harry Svensson Jan 24 '18 at 13:25
• It's a matter of style but "128 slices plus 512 interconnect lines" is not that much relative to the resources available in the spartan and (in my opinion) optimising it probably wouldn't be worth the effort if area isn't a huge concern (is it?). I would code the simplest implementation and come back to it if area becomes an issue. – stanri Jan 24 '18 at 20:14

As @Neil_UK said, you seem have answered your question, and are looking for confirmation here. Whatever we say and advise here, responsibility with your end product remains with you.

Second, it sound that you have been charged with design of FPGA/SoC part of the project, and there're other participants taking care about processing elements. If you would be in charge of whole project I would expect seeing some words how PEs will look like for paralle and serial communication, and some calculations how many wires PEs will need if connected with parallel interface (16 wires * 64 elements = 1024 wires?) and how many multiplexers will be needed. If I am correct, then it is necessary to synchronize between other developers within your project on the solution architecture before going to implementation and optimization of your part.

Third,

Each PE outputs two 16-bit words of data (A and B) at a frequency of ~100 KHz. ... The system clock will be >100 MHz, so I have >1000 clock cycles to collect the data from the PEs.

Let's calculate how much time design will spend if there will be only single serial input (e.g. SPI) for 64 elements: 100 kHz * 16 bits * 2 words * 64 PEs = 204 MHz (with zero protocol overhead), thus you will not make it with 100 MHz clock this way, you must first devise the architecture(s) which can handle requirements before thinking about optimization.

And as you understand from your own question - there's a trade-off between having complexity in FPGA (e.g. logic elements, registers, interconnect, internal clock speed), and outside of FPGA (number of wires from the PEs). More wires from PEs mean you will need less work at FPGA side, and vice versa.

Update (for question update and below comment):

It is still not clear about architecture of your solution. Seems PEs are located within the same FPGA? I am afraid there's no right answer to your question. I suggest you try designing both ways - parallel and 1-bit serial, and see the difference in the FPGA resources used. Here's why:

1. parallel will use small mux at PE side anyway, to choose between 2 words (switching RAM address); serial will have some more logic at PE side serializing these words, add here some control logic and selection of the currently running PE from the data reception side;
2. most compilers are having built-in capabilities to optimize hardware structures they create, thus optimizing things yourself you may confuse compiler and it may start performing stupid things when optimizing, or even stop optimizing at all. Simpler and more uniform code is, better compiler can optimize.

From my perspective, synchronous serial communication (1-, 2-, 4-bit) must win in terms of resources used and power consumed, but will lose in terms of performance. But you need to experiment.

• It's a private project with nobody else involved. Each PE operates on a block RAM. The results A and B are read as 16-bit values from the RAM output port (which cannot be any other width). It's all in the same clock domain. Your parallel wire calculation is correct. If serializing from 2*16 bits down to 1 bit is too slow, I could serialize down to 2 bits I guess. But is serialization the way to go at all? (Please see my update to the question) – travelboy Jan 24 '18 at 9:44
• Updated answer. – Anonymous Jan 24 '18 at 10:06