Best board-to-board low latency protocol/interface

Question

I've been asking similar question which received some downvotes, so I'm reformulating my task (hopes this one gives more clarity).

I'm planning to establish a high-speed (3 Gpbs full duplex) serial connection between two fpga boards (distance between the chips < 15 cm, including the connectors), more precisely between spartan 6 and artix 7 devices (these are not absolute requirements, but appear to be the cheapest ones having integrated transceivers inside).

I'm not sure about what protocol should I pick. I've been digging into RapidIO specs lately, but it looks like I have to use a switch chip to connect devices, but switches are pretty expensive (~ $150/piece) and introduce at least 100 ns delay, as well as 10+ layers PCB requirement (example).

PCIExpress is not an option (though connector is nice), since it introduces around 500ns delay, and ideally I'm looking into sub 100ns transmit latency (from parallel stream inside one fpga to parallel stream inside another). Due to high latency 10Gb Ethernet is also not an option.

There's also an Aurora protocol from Xilinx (link), but I cannot find any PHY layer definitions and delay numbers. More of that, despite all high-speed serdes applications advertise low pin counts, all dev boards with embedded transceivers use either SMC, optical or cryptic mezzanine card connectors with 400 pins. Is there any other option? Like, ~50 pin connector or less. So, to sum this up, questions are:

1. Is it possible to achive sub 100ns delay between two chips using serdes?
2. Are there any PHY specs/examples for Aurora?
3. Will it be simpler to throw wide BLVDS interface and use high-clock parallel bus?

Any suggestion is greatly appreciated!

Answer 1

Aurora looks like a reasonable choice; basically it looks like the thing you'd build if you were trying to build a generic interface from high-speed SERDES. It has flow control. It even has channel bonding, which seems to have some support in Xilinx hardware. That should give you your required speed from theoretically a single link pair in each direction at the 3Gpbs max.

This link gives some latency numbers, for a loopback link. Board to board latency will obviously depend on wire delay.

It shouldn't require a more complex stackup than the FPGA itself (usually minimum 4?). I would suggest using the PCIe connectors even if you don't use the protocol; alternatively SATA connectors might provide a good short cable option.

They don't actually specify the PHY at all, but this app note gives an example of how to connect two eval boards. They're just wired up with mini coax leads using SMA connectors, so I guess the required impedance is the usual 50 ohm?

Answer 2

You should definitely use the gigabit transceivers. With Spartan 6 you can send up to 3.125 Gbps using one transceiver so that is no problem.

These transceiver works very well and are very easy to implement.

If you have SFP connectors on both boards you only have to buy one SFP copper wire cable to connect the two boards. If you have SMA you only need to buy four SMA cables. There are also SFP to SMA cables.

When I have done these kind of project I usually define my own protocol however you need to send some K-words to align the bytes in the serial stream. You also need to define an idle sequence for the elastic buffer. Usually I use the same K/D as defined for Ethernet.