Measurement of relay characteristics so can detect before failure

Question

I am working with a very large safety system which was designed around 30 years ago. It has a system of relays interlocked together in a logic configuration so that it will failsafe if any relay itself fails. There are probably around 20000 relays in this system altogether, all on rack mounting cards.

The cards are in about 50 racks in groups of 20, spread out over a large space approximately 600m (2000ft) square. Access to some of the racks is difficult.

The relays are PCB mounted Panasonic S4-DC24V, to give an idea of the type.

The number of operations is low (est. less than 10 ops per day) and most of the time (est. > 95%) these relays are on. The reason for this is these cards are driven by door/hatch contacts and fixed (as in bolted and stay there for weeks) guarding.

The question- is there a parameter we could measure that could predict a failure? The obvious one is contact resistance, but the circuit prevents that (see note 1 below). An option is switching time, but I have not seen any reference to this as a predictor of relay life. Relay temperature I guess would not work either, as the contact load is low.

These cards are tested using ATE as part of a safety procedure for function. Failures are rare but are happening (see note 2), and can be a problem. You can assume replacing all the cards is not an option (see note 3), but gradual replacement based on a tested value is quite possible.

Just to add, it would make sense to have a cost/benefit analysis on cost of a breakdown versus cost of X cards and go from there. I don't have this information yet. If this affects what you suggest, you could assume a value range, i.e. 1 breakdown costs us the same as X new cards and specify X.

Note 1 - We have a ATE that checks the rack function with a few hundred tests taking about 5 minutes. In this rack, the statement is true. If we took the cards out and put them in a dedicated fixture, we can measure contact resistance, albeit resistance of 1 to 4 sets of contacts due to the routing in the circuit. If that is what we need to do, then I can put forward a case for it. I have also found a requirement that each rack should be tested every 12 months, but since the test rig was broken (my first job has been to fix it) I am not convinced this has been happening. To be fair, a new manager is concerned about restoring a correct system so that is why I am here.

Note 2 - I haven't got detailed info on the failure mode- it gets recorded as "the card does not work".

Note 3 - The facility is used 24/7 with short shutdown periods. The department is understaffed so things are reactive.

Note 4 - The system may be replaced or partially replaced in about 4 years, this means we are trying to avoid changing large amounts of it.

EDIT 2023-09-18 - 3 removed cards has been found. The test rig indicates they switch late (i.e. >200ms, but do switch), so they are not welded contacts. Welding is unlikely as the contacts are limited by fuses to <1A when the relays can switch much more that that.

Answer 1

One useful technique is to measure absolutely everything, and then to see what correlates with failures (assuming that whoever is doing the maintenance logs stuff properly).

As a specific example, a difference in release time depending on the current flowing through the contacts might indicate that their tips are degrading and likely to weld.

Answer 2

The question- is there a parameter we could measure that could predict a failure?

Have you looked at the simplest: age.

I did some work on the statistics for a system with approx 1,500 pumps, each operating about 8 hours/day. (Different in that this system had negligible safety impact.) We had individual data on each pump; after something like 6 years we had a very good aging curve.

We're now on about the fourth wave of replacements, which we do at an age where the pumps start failing at a much faster rate. From memory we picked something like "point at which failure rate is 10x the base rate." We picked 10 because there was a knee there. Certainly it was arbitrary, but it was also pretty effective and cheap to do.

If your relay cards aren't all the same age, perhaps you pick some age at which you replace the oldest ones.

---

EDIT to add: the real problem with this is that "remaining lifetime under normal operating conditions" has a very high variance. So you need to swap out quite a lot to get the benefit. And in OP's case, failure propagation is bad in that one failure stops the whole system.

Answer 3

Usually relays fail because the contacts wear out. It's strictly a function of the stress on the contact (voltage, current, inductance being switched, inrush current etc.) and the number of operations. These are sealed type so they tend to contaminate themselves over time if they are called upon to switch much of anything. The seals are also not perfect and can leak external contaminants (the datasheet explicitly warns against silicone, so any conformal coating had best be something like acrylic).

So if those parameters are known perhaps you can preemptively swap out some of the relays for new ones. It sounds like you have a large enough sample to gather data even without detailed analysis, especially if the circuits are repetitive.

A more "high tech" approach would be to try to detect the arcing EMI signature when the contacts don't close completely. While perhaps an interesting project, success would not be assured and the odds would depend heavily on the details of the application.

Answer 4

I would think you could do a continuity test to see if the relay is working and to also test if it's switching before use. AFIAK it is the contacts that fail, and the coils should remain intact if not stressed beyond their rated limits.

One thing that the datasheet says is the relays will develop NOx in a humid environment if there is arcing when the relays are switched, so I would try and keep the humidity low.

Answer 5

You say that you can't measure contact resistance in the circuit (I suppose because it has to operate permanently?)

Do you know the current trough the contact (or are you able to measure it with a clamp probe on cables)? If so, you can simply measure the voltage across the relays, and deduce the resistance.

Or if you have cards you know to be good (or the same card you suppose to be OK for now), then you can just measure voltage and compare later on to detect variations.

NB : depending on the relays, and the current, voltage across the contacts might be quite low, so you might need a voltmeter with sub-mV resolution

Answer 6

Another method used to check relay operation is to test operation/characteristics at less than their rated voltage.

A system with thousands of relays was used at a major airport and the various buildings had battery backup power supplies. These power systems were built with mains power supplies that trickle charged the batteries i.e. the batteries were just 'floating' in normal service. Catch was nobody was performing an adequate level of routine matenance on the batteries. When the the power supplies in one building were switched off the battery voltage in that building started to rapidly sag in minutes. Needless to say some marginal relay circuits failed as the voltage dropped. At that point the actaual panic was over the state of the batteries.