High speed memory interface between 2 FPGAs (Virtex 6)

Question

I have a board with 2 Virtex 6 FPGAs, which are connected to each other through 64 parallel IO lines that can operate up to 400 MHz. One FPGA, let's call it B, also has 2 GB of DDR3 memory connected to it. I also need high speed access to this memory on FPGA A (which doesn't have any DDR3 connected to it). Eventually, the plan is to cache stuff in BRAM on FPGA B and transfer it over to FPGA A.

So, I'm trying to design a high speed interface between the two FPGAs to do just this, but I don't know where to start. I am thinking of FIFOs to buffer the address and data request.

Do you know of any interfaces/protocols that would fit my application? If you can direct me to any papers, that would be great as well.

Answer 1

1. Implement a DDR controller on FPGA B.
2. Attach the DDR controller to a shared memory interface controller.
3. Attach shared memory interface bus A to FPGA B internals.
4. Attach shared memory interface bus B to FPGA B I/O pins.

You may need to make some compromises, of course - 32 bits for data, 31 bits for address, 1 control line probably isn't going to work; you really need more like 70 I/O. If you can live with 16-bit data then happy days.

Answer 2

As it's Xilinx, you could look at using Aurora to interface between the FPGAs - you'd have to implement your own memory access protocol over the top of it, but it allows you to easily get very high bandwidth between chips using the inbuilt SerDes (GTP) pins. It will handle all the lane matching and channel bonding and save you from the pain of trying to route (another) high-speed parallel bus between the FPGAs.

Answer 3

Physical layer

Are the 64 GPIOs all you have or do you have any other connections between the FPGAs? As indicated by Martin Thompson, for bandwidth you'd be better off using high speed serial connections if available.

Assuming your original post contains all the relevant data and you only have 64 GPIOs then you'll need to think about how you're going to achieve the stated 400MHz target frequency. Even though that might be the datasheet value there are many things that can prevent you achieving this frequency - see this related answer: Max switching frequency of GPIO pins of modern cheap FPGA

You also have to decide whether you want to potentially lose cycles by turning the bus around (i.e. having a bi-directional data bus) or waste pins/bandwidth by separating out the bus in each direction. Clearly it's going to be simpler to implement if pins are either dedicated inputs or outputs so if you can satisfy your bandwidth requirements in this way it will be much quicker to implement.

Protocol Layer

One option would be to emulate an SRAM or QDR device, however you'd need to add something to compensate for the variable latency. I expect the best option would be a simplistic memory mapped interface - either AXI-4 lite, Avalon or Wishbone, take your pick. You can then use standard components for buffering requests and responses.

Answer 4

If you cannot connect the FPGA A to its own bank of memory, then I would venture to say that the 400 MHz GPIO lines between the two are your best bet. Using them in basic SPI configuration or something.

If the board is routed properly, you could attempt to do some sort of PCIe communication between the two but that is very circumstantial.

You could venture into trying to do high-speed USB between the two but that would require implementing an OTG stack on both of them.

Answer 5

You can also try using the linkport communication between the two (You might have to manually right the code for this). I implemented a 4 bit linkport comm and ran it at the rates of 1Gbps.