What is the best way to transmit analog signals between 100 Hz and 100 kHz over a distance of up to 6000 m with minimum frequency depending loss, the level is 50 uV to max 1V (always less than 100 mV above 10 kHz). I consider using a coaxial cable (50 Ohm or 75 Ohms) to route the signal. Shall the cable be considered a transmission line? Should I use impedance matching in both ends?
I'd consider doing it digitally because the cable losses won't affect the analogue signal. You'd need 16 bit resolution - this gives a resolution of 15 \$\mu V\$ in 1 volt and you'd need to sample at 250k Sps to give your DAC filter a chance to reproduce 100 kHz.
Keeping it as tight as possible means probably a 20 bit word where the first 4 bits are a constant header so that you can recognize where the MSb is going to be. You could just about do it with 18 bits but 20 bits will play safe.
So it's 20 bits sent 250,000 times per second - that's a data rate of 5 Mbps - how far can you get this down decent coax? The sort of coax I've used in the past has transmitted 40M bpsec for 350m so maybe at one-eighth the data rate you'd get 8 x 350m = 2.8 km. The coax was 12mm diameter stuff and 75 ohms but I can't remember the part number - I know I went for the "fattest" coax I could fit in the cable glands and hoped it would be OK (and it was).
Looks like a couple of repeaters would help at 2km and 4km. Screened twisted pair might be even better than coax. Try looking up twinax.
Sending it analogue will be a problem unless you are prepared to suffer degradation of the signal in the frequencies above 100 kHz as the cable characteristic impedance changes from resistive-capacitive to purely resistive - you could try pre-emphasis and de-emphasis to help. Noise will surely be a problem over that distance and maybe someone else who has tried this can offer some advice. External pick-up is an unknown quantity but at least for digital you have a threshold that noise has to exceed.
Yes, I suggest using the transmission lines if the transmission frequency is above the 300 Hz. Theoretically, considering the line to be lossless, the propagation velocity in 50 ohm cable to be about 1.8E8 m/s and keeping the electrical length to less than 0.1 rads (your beta*length) over the distance of 6000 m, you can safely avoid the use of the transmission lines. However, in reality your line is lossy! It would also depend on your attenuation constant of your cable. Higher the frequency, higher the attenuation as the resistance of the copper is affected by the frequency. you basically want your electrical length to be less than 0.1 to avoid interference so, summing all up, I would say, you won't need to use the transmission line if your frequency is below 300 Hz; otherwise you must use matched load transmission line structure.