FPGA: efficient way to collect data from many cores within SoC

Question

I am looking for an efficient (in terms of footprint) way to periodically collect data from a multitude of processing elements (PEs) within an Spartan6-based design.

I currently have 16 PEs, but when the design scales up it may become up to 64. Each PE outputs two 16-bit words of data (A and B) at a frequency of ~100 KHz. The output data from all PEs needs to be collected and added to two sums (A_sum and B_sum). The system clock will be >100 MHz, so I have >1000 clock cycles to collect the data from the PEs.

The easiest way to do it would be two 16-bit wide multiplexers (A_mux and B_mux) with one input per PE each. For 16 PEs, this would cost 128 slices plus 512 interconnect lines within the chip, and it would take 16 clock cycles to collect the data. Since it doesn't need to be that fast, I am wondering if there is a more resource-efficient way to do it.

I was thinking of serializing the data in some way. This would save interconnect lines, but I would use more slices to implement the serializers in every PE. I would also need shift registers with parallel outputs (hence no SRL32) for de-serialization.

UPDATE

Some of you suggested I may have answered my own question. Serialization obviously comes to mind, however I am not sure whether it is really going to be more efficient. My question is not so much whether serialization is feasible, but I am concerned that serializers for every PE will use up more FPGA resources than a 16-bit multiplexer with inputs for every PE. Maybe there is some clever implementation I haven't thought about, but it seems to me that the fat mux is really the most resource-efficient way. Am I wrong?

Answer 1

As @Neil_UK said, you seem have answered your question, and are looking for confirmation here. Whatever we say and advise here, responsibility with your end product remains with you.

Second, it sound that you have been charged with design of FPGA/SoC part of the project, and there're other participants taking care about processing elements. If you would be in charge of whole project I would expect seeing some words how PEs will look like for paralle and serial communication, and some calculations how many wires PEs will need if connected with parallel interface (16 wires * 64 elements = 1024 wires?) and how many multiplexers will be needed. If I am correct, then it is necessary to synchronize between other developers within your project on the solution architecture before going to implementation and optimization of your part.

Third,

Each PE outputs two 16-bit words of data (A and B) at a frequency of ~100 KHz. ... The system clock will be >100 MHz, so I have >1000 clock cycles to collect the data from the PEs.

Let's calculate how much time design will spend if there will be only single serial input (e.g. SPI) for 64 elements: 100 kHz * 16 bits * 2 words * 64 PEs = 204 MHz (with zero protocol overhead), thus you will not make it with 100 MHz clock this way, you must first devise the architecture(s) which can handle requirements before thinking about optimization.

And as you understand from your own question - there's a trade-off between having complexity in FPGA (e.g. logic elements, registers, interconnect, internal clock speed), and outside of FPGA (number of wires from the PEs). More wires from PEs mean you will need less work at FPGA side, and vice versa.

Update (for question update and below comment):

It is still not clear about architecture of your solution. Seems PEs are located within the same FPGA? I am afraid there's no right answer to your question. I suggest you try designing both ways - parallel and 1-bit serial, and see the difference in the FPGA resources used. Here's why:

1. parallel will use small mux at PE side anyway, to choose between 2 words (switching RAM address); serial will have some more logic at PE side serializing these words, add here some control logic and selection of the currently running PE from the data reception side;
2. most compilers are having built-in capabilities to optimize hardware structures they create, thus optimizing things yourself you may confuse compiler and it may start performing stupid things when optimizing, or even stop optimizing at all. Simpler and more uniform code is, better compiler can optimize.

From my perspective, synchronous serial communication (1-, 2-, 4-bit) must win in terms of resources used and power consumed, but will lose in terms of performance. But you need to experiment.