Postmortem: on a PDU with low Power Factor, why did all of our servers reboot?

Question

I am a Unix SysAdmin with a crude understanding of electricity.

We had this scenario: 4 dual-PSU servers were plugged into a ServerTech Sentry Switched PDU with PIPS with Y cords. The circuit is 208V/30A. Circuit load is under 1A, so that is 8 server-class computer PSUs drawing around 0.1A each. My colleague plugged an additional server (older hardware redeployed) into the PDU. While provisioning this system, he discovered that the four lightly-loaded systems had all rebooted.

We began investigating. This is when we found the Power Factor (PF) on the PDU was 0.75 "WARNING LOW" ... we unplugged half the server PSUs and the PF improved over 0.8. I filed an RMA with the server vendor for our defective PSUs. The vendor then reported to us that PF is low at low load, and that if we want to improve PF we should run the systems at higher load. Running a stress command on one server brings the PDU PF over 0.9.

The question we ask ourselves is:

Is it reasonable that under these circumstances, that plugging in an additional system should trigger a state of temporary power failure on the low-load systems? Or, does this imply that the PDU or the server PSUs are defective?

As best I can make out, we exposed an edge case for our datacenter practices, and we should make a move to monitor PDU PF, flag when it is low, and take corrective action (like artificial load) when a situation like this occurs.

Addendum:

At Week 29, we had 4x systems with dual PSUs on the circuit, and we connected a new server, which is when the 4x systems rebooted. At Week 31, I disconnected half the PSUs to correct the PF warning. I will introduce PF instrumentation in the future.

Answer 1

pF is picofarad.

Now, Power Factor is misunderstood...

Mains voltage is a sine wave. If you plug in a resistor, like a heater, it will draw a sine wave current which is in phase with the voltage, this means current and voltage are proportional. This is the ideal situation for the utility, as it minimizes losses in wiring, transformers, and other distribution equipment from the powerplant to the user.

Other kinds of loads will draw current in different ways. For example an inductive load, like a huge induction motor as used in industral machinery or elevators, will draw a sinewave current, but it will have a phase shift relative to voltage. Older switchmode power supplies used in computers contain a rectifier followed with capacitors, so they will only draw current on the peaks.

Both cases are far from optimum for the power utility, as your current waveform causes higher losses in their network. But they only bill you for the power you use, not the losses in their network. So they will want you to pay extra to recoup the lost power.

"Power Factor" is a single-digit measurement of how current strays from the ideal sine wave. It is not perfect, but it is useful.

SMPS computer power supplies convert a high DC voltage like 320 VDC into low voltages like 12V, 5V, 3.3V, etc. Current technology can do this very efficiently. The problem is how to get the 320VDC. You can use a rectifier from mains and a capacitor, but this will only draw power on mains voltage peaks, and have a bad power factor.

Thus, a PFC (power factor correction) circuit is added. This is a boost converter which takes as an input the rectified mains voltage, and generates high voltage DC, while drawing power from the mains in a well-behaved sinewave fashion.

However, this is not 100% efficient. If the circuit is designed to be 95% efficient at full load, then at 10% load it will lose efficiency. It is reasonable to expect the manufacturer would make a compromise, and take a loss in power factor in order to keep efficiency high. In fact, at very low load, when the PC is in standby and draws like 0.1W, the power factor correction would most likely turn off entirely, as its losses would exceed the actual used power.

Anyway.

Power factor measures how much evil the system does to the mains and the utility, but you have to remember it's a factor. The actual amount of harmonics and badly-shaped current drawn from mains depends on how much power and current the load actually uses. Thus, a load with bad power factor, but which uses little power, is not a problem. The problem is loads which use lots of current AND have a bad power factor.

Is it reasonable that under these circumstances, that plugging in an additional system should trigger a state of temporary power failure on the low-load systems? Or, does this imply that the PDU or the server PSUs are defective?

No, this is not reasonable.

When you plugged in the additional system, it drew an inrush current from the PDU. Maybe the inrush current was too much, maybe not, who knows. If the PDU can't deal with that, it is the PDU's problem. If it cuts power to the other outlets when one single outlet misbehaves, it is also the PDu's problem.

It looks to me like your PDU is crap. I mean, did any other servers in the datacenter but the ones on this PDU crash? Probably not.

take corrective action (like artificial load) when a situation like this occurs.

An artificial load (like a resistor) draws power. This raises the power factor (by making the misshapen current drawn by your computers smaller in comparison) and it also raises your electricity bill. It is more eco-friendly to simply donate money to your utility.

Answer 2

Agree with comments relating to unlikely cause being poor Power factor. My following experience may be relevant. I have also conducted an investigation into stress testing, that was causing occasional PDU failure. This stress test was to simply switch the relay controlled outlets of the PDU On an OFF within 6 seconds. On this occasion failure was due to damage to power relays in a dual supply PDU's. The contacts of the relays in question are supplying the 240vac to outlets on the PDU than power server smps, typical in data room racks . The problem was these relay's contacts were occasionaly welding together when the loaded PDU outlet was switched on by operating the relay . The relay contacts were found to be adequately specified to cope with the load they were switching. After much questioning it was discovered that only the servers powered by Dell model N0. D1100E-SO smps ( A reviewed as a reliable and good quality item) were causing the contacts to occasionally weld. I took a sample of this PSU to an EMC lab to check the inrush current. Briefly, the results showed that the Dell PSU did not appear to exceed the Dmax limit of 4% and that it may comply with EN61000-3-3 . A good Dmax figure may also imply that high instantaneous inrush control is implemented within the power supply. So no clue there. I then did some environmental tests on the PSU / PDU combination to try to replicate the problem - and low and behold when this PSU's was only operated in an environmental temperature greater than about 21 dec C the relays would occasionally stick to a permanently ON condition VolLa!! . Opening up the Dell psu I found the classic Thermistor type inrush limiter configuration on the mains input. The thermistor was placed out of the cooling fan air stream. In an ambient above 21 dec C,the stress test period of ON - switch OFF - wait -6 secs - ON was therefor not allowing the thermistor to cool sufficiently to allow inrush current limitation to an acceptable level - hence the contact welding to occur. What is the relevance of this to the question? While this problem was revealed under "stress" conditions, be aware that high Inrush current can introduce fast transients (flicker) on to the mains supply to other equipment -quite possibly causing a reboot of the other equipment sharing the same PDU . Some of the more basic Inrush protectionof smps is not always effective in limiting this kind of interefrence - because as revealed here,the EMC. test's do not address all senarios.It is therefor quite possible that the "additional server (older hardware redeployed)" your colleague plugged in had little or no inrush current limitation. Hope that is of some value -

Answer 3

• modern PSU's are high PF with some "Green" rating, your older one may not

• the PF may be an indicator of another root cause; conducted EMI , radiated EMI, load regulation

• since each PSU has a line filter, but may be sensitive to step or impulse loads and a surge may occur on connection so depending on output impedance of PDU it could be at fault to capacitive step loads

PDU specs?

I would also explore inrush current rise time on older unit and look for radiated crosstalk that cause a system reset. Scoping radiated noise on reset line, and examine system log for cause of reboot.

It is an abnormal transient caused by power noise, but where/ how was it switched on? PF should never be an issue for low power use.

Another test is add a dummy load (300W halogen bulb PF=1) transient test on/off repeated with long unshield separate wires to create a loop antenna effect for radiated noise. and also conducted surge current on PDU. It should pass this test , otherwise you have a problem.

Once the fault root cause is known, the solution is easy. Filter, shield, shunt cap on PDU, etc.

I recall almost 20yrs ago at our server factory, dual PSU's under current sharing were unstable (oscillation ripple) without at least 10% loading on DC. This could be triggered by an AC transient.

Answer 4

This actually does not sound to me like a PDU issue.

I would be looking at the measured supply impedance at the input to that PDU, it is possible you have a loose screw or similar (Cee form connectors are notorious for this). When you plugged the extra server in, it imposed a transient current demand as it charged its power supply caps, which could have pulled the line voltage down for long enough to cause the reboots. Do you have a log from the PDU showing any voltage excursions around the time of the reboot, I would be looking for a honking great dip that comes back over a cycle or two, is this reflected in any other PDU logs?

Modern PC power supplies (Yes even most server supplies) are junk with very marginal hold up times (20ms on a GOOD day), because doing this right costs money and space.

PSC is largely irrelevant with lightly loaded supplies, I would be logging current, kVA, kVAr, Voltage (Including logging any deviations), and would like to have available (if not logged) current harmonics.

Incidentally, if those were my servers I would be very tempted to virtualise the lot onto one box that would then be at least slightly loaded, better for power efficiency, frees up space in the racks....