17
\$\begingroup\$

I have implemented a design that uses PCIe. It is somewhat different in that the PCIe interface is used as a chip-to-chip communication lane on a single PCB (e.g. no PCIe connector).

The root complex device is a Freescale i.MX6 which is PCIe Gen 2 compliant and the device I am communicating with is a Marvell WiFi module that is a PCIe Gen 3 compliant device. It's a single lane interface running at 2.5Gbps.

I've done some signal integrity measurements by soldering a high speed scope with proper differential probles right on the other side of the inline caps shown below:

enter image description here

For the clock the eye diagram looks quite good:

enter image description here

But the TX data not so much:

enter image description here

The WiFi chip has on-chip terminations so I don't believe I am supposed to need any additional terminations, but I could be wrong about that.

I have found some registers that can be set within the i.MX6 processor for the PCIe peripheral but I'm not exactly sure what they actually do. A little bit of trial and error hasn't gotten me very far either.

enter image description here

I've checked that the layout follows proper routing rules and the PCB was constructed with the correct impedance. Obviously I have some jitter in the system but it also looks like I have a reflection or de-emphasis issue. I'm hoping someone could describe what they see wrong with my eye diagram and/or suggest some ways to fix it.

Cheers!

\$\endgroup\$
  • \$\begingroup\$ cool question. Hope to see some good answers. \$\endgroup\$ – justing Jul 14 '15 at 1:22
  • 1
    \$\begingroup\$ Where are you measuring your tx eye? \$\endgroup\$ – Some Hardware Guy Jul 14 '15 at 2:14
  • \$\begingroup\$ And what are this registers set to? \$\endgroup\$ – Some Hardware Guy Jul 14 '15 at 2:18
  • \$\begingroup\$ A couple questions: which TX is the problem? the i.MX6 to Marvell or Marvell to i.MX6? What kind of supply filtering do you have set up around the misbehaving transmitter or its clock supply rails? What are the current values of those registers? \$\endgroup\$ – akohlsmith Jul 14 '15 at 2:21
  • \$\begingroup\$ @SomeHardwareGuy I am measuring the tx eye right on the other side of those capacitors circled. In terms of the layout the capacitors for both the clock and the TX are right next to the WiFi chip. Total distance the pairs are routed is about 2.5 inches. The i.MX6 is actually part of a SOM that plugs into an SO-DIMM connector and I've only designed the board that "carries" this SOM and the WiFi chip. \$\endgroup\$ – Funkyeah Jul 14 '15 at 7:05
22
+50
\$\begingroup\$

There are quite a number of things that will do this to you.

You have not stated the length of the interface. I do direct chip to chip PCIe frequently and you really need to take this into account as you will get attenuation of roughly 0.18dB per inch due to skin effect losses and about 0.5dB per inch due to dielectric absorption on 'ordinary' FR4.

I think you may be able to get better numbers from the PCB material datasheet if you download it and look at the loss tangent. Take a look at Isola 370HR for a typical datasheet. The numbers above are pretty accurate at the 5GHz rate. At the 2.5Gb rate, the numbers are a bit lower, with a total loss of ~ 0.4dB per inch.

I am assuming that apart from breakout and the coupling capacitors, you are using single-layer routing for the interface. Layer transitions can easily do very nasty things to the signal. Controlled impedance will be a bit different layer to layer and reflections are the natural result (there are ways of successfully doing multi-layer routing, but it takes a great deal of care and some unusual tricks to achieve).

For PCI express (and Infiniband for that matter), the rise and fall rate of the signal at the transmitter has a minimum rise and fall time to minimise EMI issues, and that time is 0.25UI, which yields 10GHz signalling artefacts on gen 2 links and 5GHz artefacts in gen 1 which must be taken into consideration.

The de-emphasis field above helps you get a clean eye at the receiver by bringing the non-switching amplitude down relative to the nominal launch amplitude.

If you are losing too much amplitude at the switching edge of the signal, set this field to a larger value. You might also set the nominal launch amplitude a bit higher as well.

Other issues you may look at:

Where, relative to the transmitter, are the coupling capacitors? They should be as close to the transmit pins as possible. Once they are more than half a wavelength of 10GHz (about 0.6 inches on FR4) [double that distance for 2.5Gb/sec links], they will most definitely reflect energy.

I have had problems with capacitor geometries of 0402 or larger in PCI express gen. 2 and I now use reverse geometry devices (0204) for the reduced effective series inductance. These seem to be getting the job done very well.

Looking closely at the eye diagram for transition bits (nice scope you have, by the way - that is what you need for this stuff), the initial signal drive appears to be driving a terminated line (the signal goes to 0.5 V(nominal) in the classic transmission line manner). This is not a particularly long line (the round trip time is where the signal drives to full V[nominal]).

you say that the link is running at 2.5Gb/sec (gen 1) and that is what the scope traces show, but you may want to experiment with the de-emphasis field (look in the reference manual as well as the electrical data sheet (see note below). The 'standard' de-emphasis values are for a nominal link, not an embedded link such as you have here (and what I also do regularly). If you can get it to about 6dB, you may get better results.

The nominal de-emphasis is 3.5dB for Gen 1 and 6dB for Gen2. I note that the field above seems to imply that a Gen 2 link is 3.5dB - you may want to dig a bit on that. The link you have needs a minimum of 3.5dB of de-emphasis.

Note that the eye at the receiver will be very different, but this is where it matters. As an experiment, read the error counters in the processor (most of them have a counter for retries); if you are not seeing significant errors, you may be chasing something that is not really a problem. If you are seeing large error counts, then maybe some of this might help.

One more thing: excessive launch amplitudes and de-emphasis are just as bad as setting them too low.

Maybe that will help you a bit: Hope so.

Note: Freescale documents everything, it is just that sometimes it is not in the place you expect to find it. Make sure you have the latest device errata as well.

Update. Added notes about capacitor geometries.

Up to the 2.5Gb/sec node, 0402 devices are fine. My handy calculator shows a typical 0402 device has about 10 ohms of impedance (inductive) at this frequency and 21 ohms at 5GHz (the highest frequency of interest). This is not too bad in a 100 ohm differential system as the effective impedance of a closely coupled pair is somewhat less than a straight addition.

The self resonance for this device is 19MHz, well below all frequencies of interest, so any phase noise is due to ESL. Keeping the impedance down to less than about 1/3 of the effective track impedance means that the phase noise (and therefore additional ISI) we will get is between 1 and 17 degrees (a single ended track from a closely coupled pair is typically (Z(diff)/2)*1.25, so for 100 diff, the single ended impedance is about 65 ohm). This amount of phase noise is manageable.

At 10GHz, the effective impedance is about 44 ohms, and can start to interfere with the differential pair by introducing excessive phase noise across the frequency band of interest as the maximum phase is now about 34 degrees. Although I have successfully done Gen 2 with 0402 devices, I have also had issues with longer runs and now use 0204 reverse geometry devices for this speed and higher.

For 8b/10b encoded links, the frequency band of interest is from bit rate/5 to bit rate * 2. The lower limit is bounded by run length encoding, and the upper limit is bounded by the specification requirement for minimum rise and fall times.

The ESL for various geometries:

0402: about 700pH

0204: about 300pH

0805: about 1nF

Update Added commentary about 50% initial launch amplitudes.

Let us consider a transmission line terminated at the source and destination at the characteristic impedance of the line, Z0.

At the initial launch, assuming the line is long compared to the wavelength of the signal, the launching point will go to 50% of launch amplitude due to voltage divider effect (the transmitter sees only the transmission line at this point).

Once the energy arrives at the destination point and starts rising to the 50% point, the energy at the source has effectively 'filled' the line with energy and rises to the full launch amplitude. Strictly speaking, the transmission line at the destination sees a voltage divider, and the divider effect at the source disappears as the output approaches DC (keeping in mind that the transmission line effect is applicable only to transitions).

This could also be visualised as the 50% energy level moving down the transmission line to the final termination and then reflecting back at full level. That is why we see a 'round trip time' in the 50% point at any point on the line.

Your plot shows precisely this behaviour at a point in the line that is not yet at the destination port, because this 50% point is in fact moving along the line.

At the receiver, once the energy has reached to 50% point, the full energy of the line is following and the voltage at the receiver continues to rise, giving a smooth transition from one level to the other.

This could also be visualised as the 50% voltage point moving along the line to the receiver, then reflecting back at 100% (the receiver achieves DC first). For that reason, the voltage at 50% viewed at any point on the line shows the round trip time from that point to the receiver.

This discussion is as valid for differential signals as single ended.

So your plot above shows classic transmission line behaviour with little excursions beyond the expected behaviour. In fact, this is one of the cleanest transmitter eyes I have seen.

\$\endgroup\$
  • \$\begingroup\$ Good answer. Maybe add a reference or some measurements where you see reflections from your caps in presence of any serious loss? I have never seen that and believe it to be more of a myth. \$\endgroup\$ – Rolf Ostergaard Jul 14 '15 at 9:03
  • 1
    \$\begingroup\$ In the comments below the post I did clarify the length is about 2.5 inches. We actually happen to be using Isola 370HR and have managed the layout quite carefully. Having said that we are also using a SOM which makes placing the caps near the transmitter essentially impossible. I am measuring on the receiver side of the coupling caps which is as close as I can get to the receiver before the routes drop into a controlled impedance inner layer. \$\endgroup\$ – Funkyeah Jul 14 '15 at 14:40
  • \$\begingroup\$ Having said the above beautiful answer. I'm going to mark it correct even if it doesn't directly lead to solving my particular issue since it has so much useful knowledge. Any suggestions for good resources on this topic? I find the actually practical knowledge is locked away in power points and the occasional white paper. \$\endgroup\$ – Funkyeah Jul 14 '15 at 14:47
  • \$\begingroup\$ Rolf: the caps will cause some reflection if they are a fair distance away, and it can be managed but it does need to be recognised, especially if the signal has to come from an inner layer to and from the caps, adding a couple of vias. The discontinuity is not huge, but if the link budget is being stretched, it can be the straw that broke the camel's back. \$\endgroup\$ – Peter Smith Jul 15 '15 at 9:15
  • \$\begingroup\$ Few more questions if you don't mind! What do you mean when you compare 0.5V initial to Vfull and mention roun-trip time...? Are you saying that the first part is is measuring the reflection and the signal and the last part is just a signal? Is this appropriate to see or bad? Anyway to remove it? \$\endgroup\$ – Funkyeah Jul 16 '15 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.