Excessive latency in inter-FPGA communication

Question

I have 4 Spartan 6 FPGAs connected through 64-bit lanes, forming a line. (That is, FPGA1 is connected to FPGA2, FPGA2 is connected to FPGA3, and FPGA3 is connected to FPGA4.)

I have split each 64-bit lane into two 32-bit lanes, one for each "direction". I then connect the 32-bit lanes to form a big loopback from FPGA1 to FPGA4 and back.

When testing this loopback (where FPGA1 drives the test) with a "walking 1" travelling every 100 clock cycles, I notice that the loopback is flaky (bit 7 doesn't go high quickly enough, it seems). When I reduce the walking 1 speed to move every 1000 clock cycles, the test passes.

FPGA1 is clocked at 20 MHz, so 100 cycles is 5 micro seconds. Does a signal really need more than 5 micro seconds to travel from FPGA1 to FPGA4 and back? What could be causing this excessive latency? (The FPGAs are on the same board, closely routed to one another.)

Answer 1

\$5\mu s\$ is a lot slower than I would expect for a naive implementation, so you may have some other issues. Check the Floorplanner or FPGA Editor to see what the routed design actually looks like. You can try adding a "pad to pad" timing constraint if you have not done so already.

But in general, asynchronous designs are discouraged for FPGAs. I am not sure what your purpose is, but for inter-chip communication, the best practice is to implement some form of clocked interface.