8
\$\begingroup\$

The following is the schematic for the relay section of the board:

enter image description here

enter image description here

VBAT: 45-56 V.
RY_REAR_MOT: Power to the front motor controller
RY_REAR_MOT_EN: 3.3 V enable signal from the MCU. On power up, this remains low. After a 30 s delay, this signal is pulled up to energise the relay coil.

Datasheets: Battery charger, Relay, Motor controller

Whilst its a 12V relay, the 30 s on start up always ensures RY_Rear_Mot has been precharged to VBAT before the coil is energised, and so the voltage differential across the relay remains within spec.

Every couple of weeks however, one of these relays is welded and can be observed with the battery throwing inrush issues (VBAT direct path to motor control capacitance on start up). With the relay cut open, the contacts can be seen welded together:

enter image description here

I have tried recreating the issue thinking maybe the MCU signal might be glitching under some unidentified condition, however, I have not been able to re-create this or get it to glitch.

The only other clue I have from operators is that it is observed when VBAT is at a low voltage and the charger is connected to the robot. The charger is connected directly to VBAT.

I would like to try and recreate this failure but was hoping the community here has any take on what could be happening?

Oscilloscope measurements:

  • 12 V coil is measured with differential probe across the relay freewheeling diode.
  • Vbat is in blue and battery voltage.
  • VMOT is the RY_REAR_MOT
  • I_RELAY is measured with current probe at the point marked in the block diagram (output of relay).

Measurement 1- triggered on coil being energised (30 s after powering on system):

test

Measurement 2 - Triggered on coil de-energised (system turned off):

enter image description here

Measurement 3 - Long capture to show the motorcontroller precharging. You can also see the relay energise after 30 s:

enter image description here

No voltage differential is observed across the contacts of the relay. I have ordered a datalogger and will try to capture events in operation but again, failures have not happened whilst motors are being spun.

\$\endgroup\$
11
  • 2
    \$\begingroup\$ What is the voltage across the relay coil? Don’t tell me 12V - measure it. What voltages/currents do the contacts see when they open? \$\endgroup\$
    – Kartman
    Commented Aug 30, 2022 at 11:23
  • 5
    \$\begingroup\$ If it has to open under load the voltage across it is certainly more than 12V. You need VBAT rated contacts. \$\endgroup\$
    – user16324
    Commented Aug 30, 2022 at 11:31
  • \$\begingroup\$ I will confirm the voltage across the relay coil. All my measurements were across the gate of the FET and the contacts. When open RY_REAR_MOT goes from 0V to 95% VBAT in under 15seconds. I haven't observed more than 1V across contacts when relay energises. \$\endgroup\$
    – Hasman404
    Commented Aug 30, 2022 at 11:31
  • 1
    \$\begingroup\$ Is there a possibility, that the voltage at RY_REAR_MOT_EN cannot rise during the 30 s because there is an active load? You could monitor the voltage across R34 using an opto coupler and lock the FET via X25. \$\endgroup\$
    – Jens
    Commented Aug 30, 2022 at 18:32
  • 1
    \$\begingroup\$ "failures have not happened whilst motors are being spun" That's great. I hope that datalogger stays on 24/7, though. Because we have no idea exactly when the failures happen, so presuming anything would be wrong. Don't discard data that might actually solve this conundrum. And also: contact voltage on closure is one thing. Contact current on opening is another. I hope you measure both! And I hope the logger has rugged inputs or good warranty :) \$\endgroup\$ Commented Aug 31, 2022 at 17:54

6 Answers 6

11
\$\begingroup\$

All switch contacts arc when opened or closed. The data sheet for the relay indicates that the contacts has a tendency to tack-weld if precautions are not taken. The contact current rating is much lower for inductive loads like a motor. The contacts will bounce so increasing that chance of welding. during that time the voltage can easily exceed the 12V specification. Operating at 57V (even if, as you say, "pre-charged" is dodgy. It provides an opportunity for sustained transient voltages that can generate sustained arcing. The transient voltages can be a damped oscillation. You should look at the switching with an oscilloscope. The data sheet warns that a resistance in parallel with the coil can increase the chance of tact-welding. The diode suppression across the coil is extending release time enhancing tack-welding. The 2N7000 may not be the best choice for this. The current is near the maximum.

  1. Place a resistor in series with the diode to allow the voltage across the inductor to rise to at least 12V during turn off. It can go higher if the FET can tolerate it.
  2. Pick a more robust FET that can absorb a transient through the body diode. Pay attention to the transient coupling to the gate through the Miller capacitance. Use an oscilloscope. Don't assume anything. This choice can allow the coil to be unsuppressed. Make sure that there is a large value ceramic between the 12V at the coil and the source of the FET to control the transient current path.
  3. Take steps to reduce transients associated with the motor with the focus on arc suppression. Use an oscilloscope to see the transients on the contacts.
  4. Use a different relay. You are operating this one out of the range of the specifications.

Clarification edit: The 12 rating on the contacts is an arcing specification. Most welders used for fabrication have a strike voltage of about 30V. During the arc, the voltage is transferred to the equivalent source resistance. The higher the source voltage is, the more current there is in the arc, raising the temperature and making it easier to sustain the arc. I think this welding in this case is happening when the contacts break and bounce. The arc starts at the tiniest gap then is maintained as the gap gets bigger. The maximum gap depends on the available voltage. The gap is designed for 12V. The circuit can provide 57V almost 5 times the spec. Also, the switch will bounce before fully retracting, so the hot arc, due to the high voltage, will have melted the surfaces. When they touch they weld. The diode across the coil will slow down the release, maintaining a longer arc and allowing more bounce events.

Edit2: When the relay opens, the motors will still demand current through the controller. This will maintain an arc longer than with other loads. You say that there is a capacitor. This may help, but its voltage will likely fall very quickly when the relay opens. I still think my comments above apply.

EDIT 3: The datasheet for the controller has no information about its internal circuitry. We can only speculate.

Measurement 1- triggered on coil being energised (30 s after powering on system):

The oscillograph indicates a 36.2A spike of current through the switch ~4ms after the relay coil is energized. This is near the limit for the switch with a capacitive load. The battery voltage drops an it appears that the motor is starting to turn. I speculate that this is the motors' starting current. The pulse starts to decrease very fast then oscillates a bit before decaying. I speculate that this is a switch bounce followed by an arc as the switch closes. This would seem the culprit for the damage. This may exceed the contact current rating from pulse to pulse.

Measurement 2 - Triggered on coil de-energised (system turned off)

The blue battery trace makes no sense. When the relay is de-energized the coil current flows through the diode, shown by the small negative voltage just after the trigger. About 1/2 way through this region a 5A peak oscillation occurs (pink line) indicating an arc while the switch is opening. The switch should be able to handle this current level.

Based on this latest information:

  1. The motor controller has speed control. Turn on the relay then ramp up the speed using the controller, then ramp down the speed to stop then deenergize.
  2. Use a better relay or replace with solid state switch, optically isolated or otherwise. Point 1 should probably always be used to control the start and stop currents.

EDIT4: @mbrig links an app note from TE indicates that the make current rating is limited to 20 ms. This is for a 12 volt system. Even though a step was taken to make the 57 volt system look like a 12 volt system, when the switch bounces, all bets are off. 57 volts across the gap will increase the gap length that will sustain the arc and decrease the duration max time to about 4ms.

Arc suppression for capacitive loads is difficult because current must be diverted around sensitive parts while maintaining a closed loop. That is why the AgSnO2 contacts are used for very high currents and capacitive loads. A short search on Google finds a lot of information on these alloys. The AgNi.15 (which is the contact plating in use)is more "weldable" than AgSnO2 which also has a higher resistance to erosion. The datasheets also indicate that these contacts should be used with suitable arc suppression, which should be added here.

So the root cause of this relay failure is: It is the wrong relay which several answers have stated.

A better solution is a FET switch providing a controlled soft turn on. The high turn on surge currents can shorten the life of the big capacitors due to internal magnetic forces.

Best I can do. Cheers.

\$\endgroup\$
15
  • \$\begingroup\$ That's a good point. I haven't yet measured how slowly the relay de-energises. I will capture this data tomorrow. Will desolder the diode for one experiment to see how much of a voltage spike i get when relay opens. Is it a bad idea to remove the diode entirely if voltage spikes are not too bad? Will add a series resistor otherwise. \$\endgroup\$
    – Hasman404
    Commented Aug 30, 2022 at 14:35
  • \$\begingroup\$ Could you explain what you mean by provides opportunity for sustained transient voltages? How come? \$\endgroup\$
    – Hasman404
    Commented Aug 30, 2022 at 14:37
  • \$\begingroup\$ The load is not inductive but capacitive (inverter). The motor would be the load for the inverter not the relay. \$\endgroup\$
    – Hasman404
    Commented Aug 30, 2022 at 14:49
  • \$\begingroup\$ @Hasman404: You should provide a schematic so we can male more focused comments. \$\endgroup\$
    – RussellH
    Commented Aug 30, 2022 at 14:55
  • \$\begingroup\$ I still don't fully follow. The circuit can't provide 57V? Surely the arc is proportional to the difference in the voltage between the 2 terminals not just the source voltage? \$\endgroup\$
    – Hasman404
    Commented Aug 30, 2022 at 15:09
8
\$\begingroup\$

Whilst its a 12V relay, the 30seconds on start up always ensures RY_Rear_Mot has been precharged to VBAT before the coil is energised and so voltage differential across relay remains within spec.

I bet you it doesn't remain within spec when the system is shutting down, though. Especially when the battery runs down and the shutdown is forced.

The only way to claim something is in spec is to measure it. And that means a data logger capturing the contact voltage non-stop in the field. If you don't do that, you can not claim about things being in-spec. And I bet you they are not. No magic to it: something must be wrong, and exceeding the relay specs is the most obvious thing you have not disproven. It's on you to disprove it.

I see no need for futzing with automotive relays here. A MOSFET will be cheaper and easier to deal with anyway. Do it right. And the relay you're using is absolute bottom-of-the barrel stuff. It doesn't belong in anything industrial, and doesn't belong in anything running on 48V either, no matter how clever the design is.

\$\endgroup\$
4
  • 1
    \$\begingroup\$ I don't agree with this. You can validate a system by design, you don't need to always measure it (and sometimes, you just can't measure it). And you can't have a data logger anyway in all your delivered system if it's not part of the specifications. I agree that using a good MOSFET in place of the relay would have been better here and more reliable. \$\endgroup\$
    – xryl669
    Commented Aug 31, 2022 at 14:26
  • 3
    \$\begingroup\$ I have to take issue with the " You can validate a system by design, you don't need to always measure it" statement. In every system I've been involved with in over 53 years in the space & military arena, the ultimate verification/validation (not gonna split hairs on the difference) of the design is measured data; quantifiable results. \$\endgroup\$
    – SteveSh
    Commented Aug 31, 2022 at 15:44
  • \$\begingroup\$ You can validate a system by design When the design is done following a process that gives a chance of a design that didn't miss anything - sure. But then we don't hear those questions. What we get to hear is stuff that is relatively basic, as far as professional design work goes. I'm sure that if JPL was designing this robot and they decided a 12V automotive relay will work fine, then it would work fine :) \$\endgroup\$ Commented Aug 31, 2022 at 17:50
  • 2
    \$\begingroup\$ To put it simply: the 12V relay was used because it would cost less. As soon as engineering time is expended failure-finding, never mind customer costs, the savings are gone in minutes. Pound foolish, penny wise :( \$\endgroup\$ Commented Aug 31, 2022 at 17:51
5
\$\begingroup\$

The relay contacts are rated for 12V, you are driving 57V (worst case) through it.

\$\endgroup\$
7
  • 4
    \$\begingroup\$ No i am not. 30 seconds delay from the MCU means the relays are never turned on until they are almost completely precharged. The 57V would only happen if i had the gates turn on at start-up and i've not been able to create such a scenario. \$\endgroup\$
    – Hasman404
    Commented Aug 30, 2022 at 11:16
  • \$\begingroup\$ Internal arcs can happen through paths you don't expect. Unless you cut one open (or get a transparent relay) you can't know how the contacts got welded. \$\endgroup\$
    – Lior Bilia
    Commented Aug 30, 2022 at 11:23
  • \$\begingroup\$ Also, since you drive a motor, what happens when you switch it off? \$\endgroup\$
    – Lior Bilia
    Commented Aug 30, 2022 at 11:24
  • \$\begingroup\$ I have added a picture of the welded relay. \$\endgroup\$
    – Hasman404
    Commented Aug 30, 2022 at 11:27
  • 7
    \$\begingroup\$ No I am not. You're blowing that relay with alacrity, you must be doing something wrong :) \$\endgroup\$ Commented Aug 30, 2022 at 17:02
4
\$\begingroup\$

Use of the PB-1000-48 battery charger makes it clear that the motors are intended for 48 V DC operation.

Your problem could stem from the following:

  1. The use of 12 V DC relay contacts to switch 48 V DC motors.

  2. Lack of arc suppression at turn off.

Here's the relay contact data.

enter image description here

The limiting 'continuous wattage' (12 V * 45 A = 540 W), 'making wattage' (12 V * 100 A = 1200 W) and 'breaking' wattage (12 V * 60 A = 720 W) may have been surpassed in your application.

\$\endgroup\$
3
  • 2
    \$\begingroup\$ As others have asked, connect an oscilloscope to pin-5 of the relay and the gate control signal and show us the oscillogram. Using a 12 V relay in a 48 V circuit is asking for trouble. When the relay contacts open up, a sustained arc will form which fries (melts, pits, strips plating, builds up carbon deposits, causes high resistance, ...) the relay contacts. 48 V can draw a fairly long arc. A snubber across the relay contacts can help, but you're better off using a relay with 48 V compliant contacts or a solid-state switch. \$\endgroup\$
    – qrk
    Commented Aug 30, 2022 at 21:55
  • \$\begingroup\$ I will test this today and try to confirm if it goes out of spec but yes based on all i have read i clearly need to find a better solution. \$\endgroup\$
    – Hasman404
    Commented Aug 31, 2022 at 9:07
  • \$\begingroup\$ @qrk i have added some scope captures \$\endgroup\$
    – Hasman404
    Commented Aug 31, 2022 at 18:05
2
\$\begingroup\$

I think you need to monitor contact current directly. Preferably without affecting it (a non-contact high bandwidth - like 100kHz- LEM current sensor, for example, is a relatively inexpensive way to get the measurement). It would be worse than useless if you added a shunt and the added resistance/inductance was enough to prevent welding.

Unless I missed something in the above, you have no idea what the peak current is when the relay contacts close. And when you add the sensor, you should not change the wire length (series resistance) or the routing (series inductance), just slide the sensor over the wire and replace the wire from whence it came. There's a small effect from the sensor itself, even with a single-turn primary, of course, nothing is for free.

If your motor controller has large low-ESR capacitors even a relatively small voltage could result in a huge peak current that could exceed the specifications and weld the contacts. As someone said, you should also wait for 20ms or whatever for the relay contacts to firmly close before commanding the motor controller.

Note also that your drive circuit, while simple, is sub-optimal for breaking large currents as specifically warned in the datasheet. If the breaking current is always very low this may not be an issue.

A low resistive suppression device in parallel to the relay coil increases the release time and reduces the lifetime caused by increased erosion and/or higher risk of contact tack welding.

\$\endgroup\$
3
  • 1
    \$\begingroup\$ The 20ms was a maximum make time specification not a minimum. \$\endgroup\$
    – RussellH
    Commented Sep 1, 2022 at 21:29
  • \$\begingroup\$ @RussellH Then make it 50ms to be safe. \$\endgroup\$ Commented Sep 1, 2022 at 22:03
  • \$\begingroup\$ I think it means that the arc must be extinguished within that time. If it takes longer then the contacts will be damaged. \$\endgroup\$
    – RussellH
    Commented Sep 1, 2022 at 23:44
0
\$\begingroup\$

When you say that you are precharging to VBat, I think there's something fishy here. What is the impedance of the motor's coil / motor controller ? Tens of ohms ?

You'll have a resistivity divider here with a high part set to 1.5kOhm and a low part that's probably an order of magnitude lower (100Ohm?). In that case, even after you've reach a equilibrium state on RY_REAR_MOTOR, you'll get 10% of VBAT on this net. That might be still too much for the relay.

Please measure RY_REAR_MOTOR voltage before switching the relay, so you can assert you're in the expected voltage range (and while you're at it, you can also measure the impedance for this circuit).

If the net is on a large capacitor (like you seem to imply), it's possible the motor controller drains the stored energy in the capacitor faster when it's driving the motor than you can replenish via the 1.5kOhm resistor. In that case, the relay will actually see more than 12V on its pins. This is the case if there is a MOSFET in parallel to the capacitor to switch the motor's coil, for example (very likely). Please provide a schematic for this net (where it goes...).

Measuring your system with a oscilloscope while it's operating should tell you more about this.

\$\endgroup\$
6
  • \$\begingroup\$ I will share scope captures later today but yes RY_REAR_MOT reaches VBAT (Maybe a diff of 0.2V). I have shared datasheet to the Motor controller but unfortunately it doesnt share much and is a little bit of a black box (off the shelf part at this point). \$\endgroup\$
    – Hasman404
    Commented Aug 31, 2022 at 14:30
  • \$\begingroup\$ When wheels are spinning the relay will always be on so the current feedign into the large cap will never be through the 1.5k as its bypassed. \$\endgroup\$
    – Hasman404
    Commented Aug 31, 2022 at 14:31
  • \$\begingroup\$ This net is connected to a terminal and from there wired directly to the motor controller VMOT. \$\endgroup\$
    – Hasman404
    Commented Aug 31, 2022 at 14:31
  • \$\begingroup\$ It's likely you're only thinking in equilibrium states, not in transient conditions. When you switch on the motor, you've a signal that goes from the MCU to the motor controller (not show in the schematic above) to tell it to run the motor. At the same time you ask to trigger the relay. If the motor controller turns on its mosfet to control the motor's coils while the relay hasn't stabilized yet, you'll see more than 0.2V across the relay pins. In that case, as soon as the relay is bouncing off, you've a resistor divider with motor coil in series to the charging resistor. \$\endgroup\$
    – xryl669
    Commented Aug 31, 2022 at 14:39
  • \$\begingroup\$ Also, even if the relay is connected, there's nothing that says the motor's coil current isn't higher than what the battery can provide. In that case, part of the current will come from the battery, and part of the current from the capacitor, lowering the potential of the ry_rear_motor net. When the relay switches off, you'll have Vbat higher than ry_rear_motor (it'll slowly increase back to Vbat via the charging resistor and the capacitor). It can be lower than Vbat - 12V (even more if the motor controller still energize the coil). \$\endgroup\$
    – xryl669
    Commented Aug 31, 2022 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.