A 50 Ω line looks capacitive when driven from 600 Ω, and so they act like a low pass RC filter. The line needs to be matched.
There are three ways you can match the line
Source matching - drive the line from a 50 Ω resistive impedance. The easiest way to do this is to shunt a 54.5 Ω resistor (exact value, 51 or 56 will be close) across the output of your ...
Your cable has approximately 100pF of capacitance per meter.
30 meters means 3 nF of capacitance if considered as one lump. In practice the capacitance is not a lump but distributed so effectively it is less.
But still that load capacitance driven with 600 ohms means you have a RC low pass filter which can't pass high frequencies well.
Another reason could ...
I have a function generator with 600Ohm output impedance. I connect this output to a digitiser channel input of input resistance 1MOhm. I connect this via a 30m BNC cable
Note - this isn't a transmission line problem because the maximum frequency of 1 MHz has a wavelength that is 300 metres
Your shield (outer) and cable inner are magnetically coupled to ...
a 50 ohm transmission line acts like a capacitor when it's fed from a 600 ohm source.
To get better performance change R2 and R1 to 56 ohms and R1 to 50 ohms
For your signal generator that would mean placing a 56 ohm resistor in parallel with the start of the coaxial cable, and changing the load at the other end to 50 ohms too.