PCIe 3.0+ PCB requirement

Question

To design a PCIe*8 carrier board for a XC7K160T module, what is the requirement for the PCB?

The 16 pairs are all adjacent to each other on the 0.6mm pitch B2B connector of the module with ~20mm span , the signals on the card edge span ~40mm. The module is planed to be placed as close to the PCIe connector as possible. But *8 still means some horizontal routing and vertical spaces for routing.

For example the vertical space is 20mm, then all signals are in a (20-40mm)*20mm area, then trace length on the carrier board won't be longer than 40mm, suppose the signal rise time is 100ps, then the trace length is several times the rise length, then impedance should matter even on this small area, and I'm not sure whether will this be a "work by luck" design with such many connections (chip to module to carrier to motherboard to chipsets).

It is said PCIe3.0 works on FR4 but is at the upper limit of FR4 https://www.intel.com/content/dam/doc/guide/pci-express3-phy-implementation-considerations-idf2009-presentation.pdf, even 10GT/s's eye won't open well.

FR4 also may have many grades, what is the requirement and what to tell the PCB manufacture?

The carrier board is very simple, can very low cost 4 layer board from small vendors be used (can they even replace FR4 with some lower grade material)?

Answer 1

Bog standard FR4, the type used as standard by PCB manufacturers is not really up to the task of Gen3 PCIe.

The weave is much too coarse which means that as the pair runs along the board, the impedance changes continuously due to the dielectric constant varying. The image below tells a thousand words. Standard FR4 is on the left.

^{Image Source}

In order to avoid degrading the signal so much, for high speeds like this we tend to use a higher quality substrate. This can range from higher quality FR4 all the way to very expensive exotic ceramic materials like Rogers.

We've had reasonable success over short distances (~10cm or so) using a comparatively low cost "High-Tg FR-4" material - this is basically still fibreglass, but the weave is much more tightly packed, as shown in the right hand diagram of the picture, which means the dielectric constant doesn't vary as much. In particular we use VENTEC VT-47, though there are many high-Tg materials out there, it will depend on what your fab house has available.

For longer distance traces or going through multiple connections such as backplanes, you will have to experiment or model to see if your drivers can cope. In this case you could go for a more expensive Rogers substrate. Alternatively an option we've used in the past is to add a PCIe Redriver IC (such as DS80PCI402), which will allow you to tune the drive strength of your differential pairs to compensate for some of the effects of impedance mismatch.

Answer 2

So, if something is sold as FR4, it's almost universally glass-fiber epoxy board. (if it was something more expensive and higher-frequency capable like Rogers ceramic board, they would advertise the fire-retardant properties on the datasheet, somewhere, not as the product name.)

When designing high-frequency transmission lines, you'll want to ask the manufacturer for RF properties. You'll usually get something like a curve for \$\varepsilon\$ over frequency, and the same for \$\tan \delta\$, the loss angle. Armed with that, you can then design your differential microstrip line to be of the right impedance, and how much signal loss you'll incur when using that over a distance. In short, you're building a microwave circuit, you sadly have to start acting as if you were.

Good news is that, yes, typical FR4 doesn't have the nicest constant \$\epsilon\$, and it even depends on direction of the glass fibers, but: you're still in the region where that's relatively benign (unless you're really aiming for the cheapest of the cheap). Don't assume that anyone builds their server mainboards out of Rogers! Yes, you'll have losses, but you know, your card probably isn't 1m long...

So, yeah, sounds like 4-layer, keeping the substrate between signal layer and ground layer thin, is a viable method. Honestly, PCIe's robustness is pretty OK.

Answer 3

A lot depends on how far the tracks need to go; PCIe 3.0 is 8GT/sec (so a fundamental frequency of 4GHz) but the bandwidth is quite high due to the 128/130 encoding so you ideally want a material that has a reasonably flat loss tangent and dielectric constant over the range of perhaps 500MHz to 8GHz to minimise ISI and dielectric absorption.

The only thing that FR-4 really means is that it is flame retardent and comes in many different forms; some materials (look at the flatness of permittivity and loss tangent) that are suitable for really high speed links have perhaps a 25% premium over more 'standard' materials but you may not need those if you adjust the pre-emphasis (sometimes known as de-emphasis, confusingly) a bit, although this may well require a few iterations to get right.

Note that what you need is a material that has a relatively flat attenuation across your frequencies of interest or at least an attenuation curve that can be compensated with emphasis settings (to minimise deterministic jitter).

You should not need to use 'exotic' materials (such as speedboard).

I would strongly recommend single layer routing for each pair (all the pairs do not have to be on the same layer as the length match requirement lane to lane is not that tight). Depending on the size of the board you may be able to route all the pairs on the surface and leave the inner layers for power / ground distribution so a 4 layer board is at least possible.

Note that for these types of signal, the rise / fall times are constrained to no faster than 25% UI so 8GHz is the maximum frequency you should see (apart from some distortion but we want that minimised anyway).

As to vendors, you need impedance controlled pairs (nominally 85\$\Omega\$ differential for PCIe 3.0) so make sure they can actually do that; Don't rely on the published PCB data - I have never seen 2 different vendors recommend the same track and gap for the same requirement as it depends on just what material they are using. I always ask the PCB laminate vendors for track / gap / material; after all, that is what they do.

So you may well be able to use 'standard' materials but without the details of track distance and material (so that a link budget may be computed) it is difficult to say in a definitive way.