# Debugging a multilayer PCB?

This is not about debugging the actual hardware/software on the board...

I had my first 6-layer PCB made recently, it's a 50mmx50mm board with the following stackup:

• Signal
• Ground
• Signal
• Signal
• Power
• Signal

The actual signal types are irrelevant to this question, the problem that I found is that the 3.3V and GND are shorted together. At first I thought that some component on the board was shorting, so I went through the entire board with a microscope looking for shorted/skewed components and could not find any. Finally I took one of my blank PCB's and found that it too showed short between 3.3V and GND.

I went through my PCB tool and could not find any shorts between 3.3V and GND, even checking 0R resistor jumpers. I can't visually see anything that is wrong.

What are some methods that I can identify the problem area? The only things I'm left with are bad manufacturing (10 board copies, 2 of which are populated), such as a shorted via. How would you go about finding the short? If it's as simple as a bad via I might be able to drill it out.

I had thought of plugging in the PCB and letting it heat up and finding the hot spot, but the on-board regulator has a shut-down feature for shorted outputs. The only other thing I can think of is x-ray, but aside from being inaccessible to me, I'm not sure it would work through a solid ground plane or power plane to see the internal layers.

How do I do a "post-mortem" analysis of multi-layer PCB's to find the fault? Should I take the board and cut it into ever-smaller pieces until I isolate a small enough section?

• if you have access to a thermal camera, then run some current through the short and look for hot spots – jsotola Apr 20 '20 at 3:56
• In all seriousness, this is why you pay the nominal fee for "100% ECT" the manufacturer uses a big bed of nails to do this all in one go – crasic Apr 20 '20 at 21:47
• If you are unable to find it using the current injection method, send me a board and I will do it for you, no charge. You can get my email at <my_SE_username> dot com. Only one condition: I can publish the results on SE. – Mattman944 Apr 21 '20 at 0:34
• On my webpage, Mattman944.com, at the bottom of the page. – Mattman944 Apr 21 '20 at 0:48
• @AyberkÖzgür - Root cause and isolation process documented in new answer. – Mattman944 Apr 26 '20 at 13:19

This is a late answer, expanding on The Photon’s answer. I realize that few people will see this now, but I can reference it when similar questions are asked in the future. Over my long career, I became known as the expert at finding shorts at the large company where I worked, so besides the shorts that I had on my own projects, other people would ask for help with their boards, I isolated a lot of shorts over the years.

The OP sent me a PWB with a short so I could document some of the nuances of the process. I am describing the process for a power to ground short, these are usually the most difficult to find because of the low resistances involved.

To inject a current into the short, solder wires onto the PWB. You need a minimum of 3, power, ground, and a reference point. The reference point will be either power or ground, start with ground since most PWBs will have more ground points to access. Don’t try to share the reference with the current injection wires, since you are measuring tiny voltages, the voltage drop on these wires can swamp out the voltage drop on the planes. A wire with current flowing will heat up and the voltage drop will drift making the isolation process much harder or impossible.

I usually put the injection points on opposite ends of the board. If the short is near an injection point, the results can be confusing. This way, the short will be far from at least one of the injection points. The reference point connection should be close to the injection point, but try not to put it where a surface trace will share the current. Or, you can have the same problem described in the previous paragraph.

In the picture, the alligator clips are attached to the current injection points, the Q-ball is connected to the ground reference point.

The voltage drop across the plane will be in the millivolt range, so you want something 100 or 1000 times more sensitive to find the voltage gradients. Ideally you want a voltmeter with microvolt resolution or better, however, a resolution of 10 microvolts might be sufficient for some cases.

A small company or hobbyist may not have a meter that is sensitive enough. But, it isn’t hard to build a 1000X amplifier. Most common DMMs have 1 millivolt resolution. The 1000X amp will give you microvolt resolution. See the bottom of this answer for more details on how to build a 1000X amp.

Using a lab power supply, set the voltage to 0.5V. If the short opens up, you want this to be low enough to prevent damage on PWBs that have already been populated. Set the current limit to a few hundred milliamps. You may need to increase it later to get a sufficient voltage gradient in the planes. I used 500 mA for this example.

To start, I am assuming that the reference point is on the ground net. Using the artwork and/or schematic as a reference, probe the ground points on the PWB looking for the largest voltage difference. Try to visualize the current flowing in the plane. Current flow in the copper planes will cause tiny voltage drops. You don’t need to probe every ground point on the board, if the voltage is getting smaller, you are moving in the wrong direction.

When you have found the highest voltage drop, this will usually give you enough of a clue to find the issue. For this PWB, the short was inside two connector mounting holes that are plated through, and connected to ground. Looking at the artwork confirmed that the power plane was not cleared out around the hole causing a short.

If the results seem confusing, sometimes moving the injection points to another location can be helpful.

To confirm, I usually move the reference point to the power net and repeat the measurements, this time probing the power points on the PWB. This PWB doesn’t have a lot of surface power connections so this wasn’t that useful.

How to build a 1000X amplifier:

I looked through my stash of parts and found the most suitable opamp, an LT1492. You need an opamp with a low input offset voltage, many common opamps will not work; since the circuit gain is high, an offset of more than a few millivolts will saturate the output. The LT1492 opamp has a maximum input offset of 180 uV. There are many other opamps that have offsets this low or better. If you have a large quantity of opamps with a high worst-case voltage offset, you might get lucky and find one with a low voltage offset using trial and error.

A little offset in the output is OK since we are looking for voltage differences. The LT1492 that I used produced an output voltage of 0.02 V with the input shorted (corresponding to 20 uV at the input).

The opamp input bias current specification isn’t as important since the circuit input impedances are low.

When you are creating a little test circuit to help you troubleshoot an issue, you want it to be as simple as possible. Don’t create another issue to troubleshoot.

I don’t recommend using a wireless breadboard. If you are inexperienced, I recommend that you build it as similar to mine as possible. I used a small perf board, a 20 AWG ground bus, and 24 AWG for the other wiring.

I recommend that you power it with batteries so you don’t have to worry about any power supply noise affecting the measurements. I used two 9V batteries with a DPDT slide switch in series. If you have some experience with opamps, you can create a virtual ground with the second opamp and use only one battery.

simulate this circuit – Schematic created using CircuitLab

I don’t have any meter probes with sharp points, as the world gets more paranoid about lawsuits, these are not as common anymore. I built a probe using a sewing needle. I epoxied the needle to a small wood stick so it is easier to handle.

• Thanks for the excellently detailed answer. Just for this whole experience to lead to a take-home lesson, what seems to have caused the power plane to be not cleared around the mounting holes? Would it be a gerber processing issue on the manufacturer's side, or should special care be taken on the design side as this was missed by the DRC? – Ayberk Özgür Apr 27 '20 at 14:32
• Also, I'm making myself a x1000 amp as soon as possible, thanks for detailing this use case. – Ayberk Özgür Apr 27 '20 at 14:34

If you have a current-limited bench supply, set it to 2 or 3 amps, and connect it between power and ground on the bare board. (Better: start with 100 mA and ramp it up to 2 or 3 A if it doesn't cook off the short)

Now probe with a reasonable voltmeter and see what part of the power net is closest to the ground potential. Your short is somewhere near there.

• I have isolated shorts in many boards over the years using this method. With a hard short and a sensitive voltmeter, my success rate is 100%. Plane shorts may need a more sensitive voltmeter than most people have. When I was working, I had access to 6 1/2 digit voltmeters that could measure microvolts. – Mattman944 Apr 20 '20 at 2:26
• This method works well. I found a stupid problem with a multilayer board one time that had a small change and the dummkopf engineer forgot to re-pour the polygon pours resulting in a short that had to be drilled out. – Spehro Pefhany Apr 20 '20 at 3:03
• An IR camera can also be a helpful tool when using this approach , even if current limited, if there is a discrete short circuit location it will glow – crasic Apr 20 '20 at 21:48
• @ThePhoton: I remember someone who, after a board was fabbed, was asked if he'd run a design rule check, and responded in the affirmative. When asked if he'd examined at the results, however, it turned out he had no idea how to do that. – supercat Apr 20 '20 at 23:01
• +1. There's also a chance that enough current will fix the issue burning the short out :) – Vladimir Cravero Apr 21 '20 at 7:26

Find a spot that is clear on all layers, such as a via that is not connected to power or ground; there should be an annular ring visible if you shine a light through the back of the board. (Not all the way around of course due to the connecting signal traces, but there should be some kind of broken donut segments visible.) There are manufacturing tolerances, but if the via is clearly far off center or one of the inner layers is badly off registration, the annular ring will be partially or completely blocked. If you have mounting holes in the corners, don't forget to check those too.

Over the years I and my cohorts have been burned by bad PCB vendors, both cheap and premium. Badly fabricated PCB's are rare, but it does happen sometimes. I once had a $1M design loss due to a local premium PCB vendor who (1) modified our design by editing the gerber files, he apparently thought the design needed lots of the WLP IC signal pins to be tied to ground, and (2) same PCB vendor technician also modified our netlist to fool his QA test into passing when it detected the shorts he introduced in what was now more 'his' design than 'ours'. I'm still angry about this years later. When we confronted the PCB vendor with our evidence, they admitted it was all their fault, and they gave us a 50% refund. That vendor is now permabanned. As a defensive measure, sometimes I leave a clear area in the corner with an all-layer fiducial mark to test layer-to-layer registration, and additionally a unique layer ID such as 1/2/3/4/5/6 to verify no missing inner layer artwork. It's hard to test for registration errors or missing layers if the groundplanes are nice and solid. Sometimes we get accustomed to getting very high quality PCB even from cheap vendors, we forget to include these kind of incoming QA defensive measures. • just reading about that makes me angry. wtf. – DKNguyen Apr 20 '20 at 3:24 • I love when a vendor pulls some sh!t like that, and then wants to refund you with credit. Like, did you think I was ever going to order from you again? – The Photon Apr 20 '20 at 3:36 • Only a 50% refund???? – SiHa Apr 21 '20 at 7:15 • I'm curious, is there any rule of conduct in EE that prevents you from sharing the name of this fab house? I'd like to avoid it myself as well. – Ayberk Özgür Apr 21 '20 at 15:44 • I didn't name the vendor (they are local to Sunnyvale CA USA) because it was several years ago. Manufacturing is about process control, and this was a process control double fault: they hired somebody who subverted at least one internal QA check, and they also screwed up their customer service response. They received a lot of bad reviews years ago, but they are still in business in 2020, so perhaps they eventually corrected their process issues. But you can't protect against this kind of problem by avoiding one particular vendor, it could happen anywhere. – MarkU Apr 21 '20 at 23:38 Sounds like the voltmeter method would be best. But you can also do a binary search. Take a bare board, and cut it in half. Check both halves for a short. Discard the half that does not have a short. Cut the remaining half in half again and repeat. At some point (surprisingly small number of cuts), you will have such a small piece remaining that you can find the short easily. If the short disappears after making a cut, then the short is in the kerf line of the saw. Also, if you have a thermal imaging camera, you may be able to find a hot spot if you allow several amps to flow through the short to GND. But I think someone else already suggested that method. • One vendor even has a dedicated thermal camera for PCB debugging - it's not that expensive and works like a charm. – Jan Dorniak Apr 20 '20 at 21:03 • This method can probably find multiple shorts more easily than the other methods. If both halves has a short after splitting, you just keep going separately. – pipe Apr 21 '20 at 10:04 • Whilst that can work, it's likely to be tricky on a 50x50mm board. – Graham Apr 21 '20 at 12:18 • If power is distributed with tracks rather than a plane, this could fail if one of those tracks crosses a cut-line twice. – The Photon Apr 21 '20 at 16:12 Use a milli-ohm meter If you have a meter that can measure fractions of an ohm (some high-end multimeters have this, as do electricians' tools, or some$10 gadgets sold on ebay as 'ESR meters' may do it), you can use it to locate the fault.

Short the meter probes together at the tip. This gives you the resistances of the probes themselves. We're only concerned with differences from this value. The meter may provide a zeroing function, or we just have to subtract this measurement from our readings.

Apply the probes to anywhere on the shorted track, one probe on each side. Note the reading and how it differs from the probe-only reading. For instance, the difference might be 80milliohms.

Then move a probe about on one side, looking for the place where the reading is lowest, let's say that gives 50mOhm. Now move the second probe around on the other side, trying to reduce this reading further. Once you've got to the minimum, the location of the probes is at or near the short.

It only takes a few minutes and is quite effective. It's probably not as easy on boards with massive copper planes whose resistance is intentionally tiny, but it works reasonably well on signal traces.

• This is easy for a 2-sided board, but this is a 6-layer board. Unless I can peel the board apart, I won't be able to use an ohm-meter to move along the internal layer traces. – Ron Beyer Apr 20 '20 at 20:58
• The problem with this approach is that contact resistance of multimeter probes with oxidised PCB pads can be significant. Using a seperate power supply to inject the current and using the multimeter only to measure voltage avoids contact resistance issue. – Peter Green Apr 20 '20 at 21:48

Low-tech method

Even if you don't have a thermal camera available (as others suggested), you have pretty sensitive thermal sensors at your fingertips.

Just run some limited current (0.5A, 1A, 5A - depends on what the board is expected to do, use a voltmeter to estimate the power going into the board, it should be some 0.2 - 2 watt) and touch-check the board with your fingers. You will find the hot spot.

Failing that, just apply more current. You will either be able to sense the heat, or burn some trace visibly. In the worst case, you will get some bulge if the fault is between the flooded layers.

• A cool variation I've seen on this is to apply power and then coat areas of the board with alcohol. It will evaporate more quickly where extra heat is being generated. – notloc Apr 23 '20 at 5:31

Another low tech method.. review your Gerber's well, or post them here for our review. Also look for rotated 45 square pads on the finished board. Diptrace exports aren't always read correctly by all vendors on square pads. I had a manufacture defect caused from this one time on a 4 layer board.

The reason is that Diptrace creates a diamond and rotates it 45, but some pcba import software doesn't do the 45 command.