I am working with a very large safety system which was designed around 30 years ago. It has a system of relays interlocked together in a logic configuration so that it will failsafe if any relay itself fails. There are probably around 20000 relays in this system altogether, all on rack mounting cards.
The cards are in about 50 racks in groups of 20, spread out over a large space approximately 600m (2000ft) square. Access to some of the racks is difficult.
The relays are PCB mounted Panasonic S4-DC24V, to give an idea of the type.
The number of operations is low (est. less than 10 ops per day) and most of the time (est. > 95%) these relays are on. The reason for this is these cards are driven by door/hatch contacts and fixed (as in bolted and stay there for weeks) guarding.
The question- is there a parameter we could measure that could predict a failure? The obvious one is contact resistance, but the circuit prevents that (see note 1 below). An option is switching time, but I have not seen any reference to this as a predictor of relay life. Relay temperature I guess would not work either, as the contact load is low.
These cards are tested using ATE as part of a safety procedure for function. Failures are rare but are happening (see note 2), and can be a problem. You can assume replacing all the cards is not an option (see note 3), but gradual replacement based on a tested value is quite possible.
Just to add, it would make sense to have a cost/benefit analysis on cost of a breakdown versus cost of X cards and go from there. I don't have this information yet. If this affects what you suggest, you could assume a value range, i.e. 1 breakdown costs us the same as X new cards and specify X.
Note 1 - We have a ATE that checks the rack function with a few hundred tests taking about 5 minutes. In this rack, the statement is true. If we took the cards out and put them in a dedicated fixture, we can measure contact resistance, albeit resistance of 1 to 4 sets of contacts due to the routing in the circuit. If that is what we need to do, then I can put forward a case for it. I have also found a requirement that each rack should be tested every 12 months, but since the test rig was broken (my first job has been to fix it) I am not convinced this has been happening. To be fair, a new manager is concerned about restoring a correct system so that is why I am here.
Note 2 - I haven't got detailed info on the failure mode- it gets recorded as "the card does not work".
Note 3 - The facility is used 24/7 with short shutdown periods. The department is understaffed so things are reactive.
Note 4 - The system may be replaced or partially replaced in about 4 years, this means we are trying to avoid changing large amounts of it.
EDIT 2023-09-18 - 3 removed cards has been found. The test rig indicates they switch late (i.e. >200ms, but do switch), so they are not welded contacts. Welding is unlikely as the contacts are limited by fuses to <1A when the relays can switch much more that that.