My knowledge around this is that the 1x/10x is an attenuation factor; if I set the probe to 10x the scope should do the math and show me the amplitude of my unattenuated signal right? Why is this reading different than the 1X reading, which to my knowledge is effectively unattenuated?
The 1× and 10× settings result in the probe presenting different load impedances to the circuit being probed (antenna). The entire reason scope probe attenuation exists is to load the circuit less (and, especially, with less capacitance) at the cost of lowering the signal.
So, if you use the 10× mode on the probe (and matching 10× setting on the scope, since your scope is not one that has automatic communication with the probe) then you are reading closer to the voltage that is on the end of the coax while it's not connected to anything (open circuit).
However, that does not mean that it's a meaningful reading. In RF circuits, impedance matters. The voltage at the end of the coax while it's open-circuit is not the same as the same as the voltage of the signal when the cable is connected to a receiver.
In order to make an accurate measurement of how this antenna and cable will operate in a proper antenna and receiver system, you need these additional parts:
- A 50 Ω BNC terminator (50 Ω because that's the same impedance as the cable you are using).
- A BNC tee connector.
- An adapter from your coax's connector to BNC.
(You can also use a “feed-through terminator” without the tee; these can give better results, but they are often significantly more expensive. Some oscilloscopes also have built-in switchable 50 Ω termination, but yours does not.)
Connect the tee to the terminator, the antenna cable, and directly to your oscilloscope's input port.
Do not use the scope probe. Do not use any length of cable between the oscilloscope and tee.
Tell the scope you are using a 1× probe (not 10×), because there is no attenuator in this setup. (Strictly speaking, there is a voltage division going on between the 50 Ω load and the 1 MΩ input impedance of the scope, but that is negligible since the ratio is so large.)
This will get you the voltage reading which most accurately reflects the voltage of the signal that will be going into a proper radio receiver connected to this antenna (which will, similarly, be matched to 50 Ω).
However, there is another caveat to keep in mind: your oscilloscope is a “100 MHz” oscilloscope. Oscilloscopes don't perfectly represent every signal up to their bandwidth limit; the quoted limit is usually (including for yours, according to the data sheet) the frequency that is high enough that signals are attenuated by 3 dB (one-half of the original power). Thus, if you are observing a radio signal at, say, 103.9 MHz, the wave displayed on the scope will be significantly less than the real amplitude.
This is not a big deal for rough order-of-magnitude measurements, but it's important to keep in mind as a limitation of the tool you're using.