I found the following BNC input output box for two channels(left side inputs, right side outputs):
The grounds of the BNCs are connected via the metallic case. I checked this with multi-meter continuity test.
Then I read the cap and resistor values and drew and simulated this interface for one channel as follows:
Could this be just a low pass filter? Anything related to common-mode? What are these 10nF caps?
It is a low pass filter.
C1 does the low frequency filtering, together with R1, making a time constant of 220uS, which gives a -3dB frequency of about 720 Hz, which agrees with your graph.
At much higher frequencies, the residual inductance of C1, due to its leads and large physical size, stop the filter working, and the stopband rises again. If a filter based on C1 alone was to be used to stop RF pickup by an amplifier, it may not work too well.
The 10nF capacitors are intended to suppress this higher frequency transmission. Being a lower value and having shorter leads, they have less residual series inductance, and so filter to higher frequencies. C2 works well for this purpose.
Unfortunately C3 does not work well, as it is in shunt with C1. At some frequency where C1 has gone inductive, C3 will form a parallel tuned circuit with the inductance of C2, actually enhancing the transmission of this frequency through the filter.
Fortunately, people are rarely bitten by this mistake. In this filter, C2 may well provide enough attenuation that the C1/C3 resonance does not cause an actual problem. Or it may be used in an environment where there are no problem transmitters at the specific resonance frequency. When the large capacitor is a lossy electrolytic, it will rarely have sufficient Q to form a resonance.
All that would be needed to eliminate the C1/C3 problem is a resistor of a few ohms between C2 and C3 to 'de-Q' their resonance. Ferrite beads are often used for this purpose when the voltage drop of an extra resistor cannot be tolerated in (for instance) power supply filtering.
