I designed the following PCB, where after testing a faced a severe (EMC related?) problem.
This board houses an STM32F103 developing board.
On the right of the PCB, there are 6 thermistors connected to measure various temperatures, which during all testing where connected directly to the PCB, without any long cabling. On the bottom of the PCB there are 7 relays, SANYOU SRD-S-112D, which are driven from the ULN2003A above them. The diode of the ULN2003A is properly connected, and there is no problem with the back-EMF of the relays. No other connections where done during testing.
While I was testing the board with nothing connected to the relays, everything performs as expected.
Then I connected a mains (220V) relay to the on-board one. The connection thus is ULN2003A=>on-board relay =>mains relay. And here the problems begun. Every time the 220V relay changes state I get excessive noise spikes on the PCB. I2C data are corrupt, data of the shift register controlling the ULN2003A are also corrupt etc. The problem is more severe when the relay is being deenergized.
The PSU, the PCB and all used instruments are properly grounded. The PSU is isolated, with the DC ground connected to earth. The on-board relays are tested, and my DMM reports OL between the contact and the coil for all. Apart from ground nothing else connects electrically the mains relay and the PCB under test.
I hooked up my oscilloscope and I got the following trace. Channel 2 is the +12V supply of the PCB. These spikes are observed 6ms after the ULN2003A output changes, so I believe this is the time the on-board relay needs to change state, and the coil of the 220V relay to start/stop operating.
These spikes can be observed to any point on the PCB, on any trace and voltage level.
What exactly is causing this? How can I combat it? How can I determine whether these spikes get to my circuit through the PSU, or through the air, and in each case which is the solution?
As I have manufactured a small batch of these boards, I am interested in solutions both to make them work and not throw them away, and solutions to implement to the next revision.
More tests can be performed, please ask in comments.
As per recommendation in the answer, I installed a 1.5KE400CA TVS on the on-board relay's contact. I can say that it helped a bit, but definitely the problem persists.
As you see on the schematic, there is a serial output, for daisy chaining more shift registers, and relays. I connected such a board on this output. It is made on a strip-board, so I do not have any schematics at hand, but it is a simple 74HC595 connected to a ULN2003A. Just the same circuit as in the main board. When doing the same test (mains relay connected, to the contact of the expansion board relay), the spikes are much more severe, and the system becomes completely unresponsive, with data on all buses (I2C, and SPI) totally corrupt.
I tried adding TVS diodes on everything (relay coil, relay contact, mains relay coil, mains relay contact), but did absolutely nothing.