Top layer of the PCB, 50 Ohm microstrip line transmitting 650 MHz/1.3 Gbps
(corrected: 1.3 GHz) rectangular pulses.
To keep good signal integrity, should I remove a solder mask ink over the top of my trace?
1.3GHz rectangular pulses.... is this actually digital data? If it is and the clock frequency is 1.3GHz then the actual frequency of the signal is 650MHz and I would recommend the frequency of concern to be 3rd harmonic which is 1950MHz. At this kind of frequency I would just make sure that you account for the effect of the soldermask in your impedance calculations and leave it.
If you actually have rectangular pulses of analog data and the frequency is 1.3GHz then I would try to preserve 5th harmonic of 6.5GHz, which is getting to a frequency where things really matter. In this case I would say maybe microstrip isn't the best structure and consider stripline. If you HAVE to use microstrip then do some simulation for the length of your line with and without the soldermask (and with the line geometry adjust for the presence or absence of the soldermask) and decide what you can live with. If you can't simulate then base the presence or absence of soldermask on the line length. For long lines (greater than a several inches) consider soldermask removal (although you may find that the nickel in ENIG, if that is your plating, is worse than soldermask would be). For shorter lines the soldermask is fine.
A couple of other things... how long is this microstrip? less than an inch... less than several inches ... more than 30 inches? I have boards with over 70 inches of microstrip with frequency content of 15GHz. I remove soldermask, and do tin plating. The nickel in ENIG (and, obviously, any other plating with a nickel barrier) causes significant high frequency loss over these long lengths. I have plenty of other designs at similar frequencies but with line lengths less than 3 inches where the soldermask and even mask over ENIG have perfectly good signal integrity (as long as geometry is correct for the presence of mask).
I am less than a year in a high frequency PCB design. So below is not my answer but advises from three different sources.
Source 1. A friend of mine with a microwave experience longer than my age
Option 1: make an opening over the top of the trace, coat it with immersion gold.
My note: I know ENIG is not a best choice for ultra high speed signals, so probably he accounted for the value of my frequency (650 MHz first harmonic).
Option 2: just remove the solder mask over the top of the trace (called an exposed microstrip) allowing direct contact with air.
Source 2. My PCB foundry
I asked them how their customers usually do for microwave applications.
Response: make a standard solder mask opening (trace width plus some standard solder mask expansion, say 0.1 um each side). Coat it as you coat all other contact pads.
Source 3. Internet
Dr. Eric Bogatin "When Accuracy Counts", Printed Circuit Design & Manufacture, May 2003:
How will Zo change from a soldermask coating the top surface? To second order, we would expect the capacitance to increase and the impedance to decrease. Using the SI6000 (note: a field solver), we find Zo decreases by about 1 Ohm / mil of soldermask thickness.
Hallmark Circuits, Inc. "Controlled impedance from the fabricators view", Rick Norfolk, p.8
Remember that soldermask, in almost all cases, will exist over the impedance traces on microstrip designs... Typical thickness of an LPI mask over the traces is .5 mil and the impedance value is only affected typically by 2 ohms.
I am somewhat reluctant to use an exposed microstrip due to copper oxidation.
So, make a solder mask opening of trace width plus some expansion, coat it with ENIG (or with another surface finish if immersion gold is not suitable for your frequency). Recalculate impedance accounting for the total metal thickness. Get desired value of Z0 (adjust trace width if needed).
PS1: For reference, at my foundry, thickness of ENIG is about 4 um (4 um nickel and 0.1 um gold).
PS2: As I understand, the problem with solder mask ink is two-fold: 1) it is not conformal (complex geometry, hard to estimate impedance, see pictures here), 2) its thickness is not tightly controlled in comparison with surface finishes.
PS3: if your trace is on an external layer, account for electroplating thickness in an impedance calculator (best to contact your fab). In my fab, if I use 0.5 oz copper (18 um), the resultant copper thickness is 45 um (-3 um copper polishing, +30 um electroplating of through holes).