I only have control over what is in the dotted rectangle.
If that is true you'll need two 25 ohm resistors in series with each feed to the 50 ohm terminator to avoid reflections (placed local to the 50ohm). The sending end impedance can be zero ohms providing the receiving end is perfect (which at the moment it isn't without the two 25 ohm resistors previously mentioned). If the receiving end were not perfect, a reflection would occur but this would be swallowed by the sending end if it had exactly 100 ohms terminating it: -
So what would a 1st reflection cause at the receiving end - the waveform the receiver receives could be put "out of shape" (digital transmissions) or changed in amplitude (for CW type transmissions). Neither will be earth shattering and probably livable with but, the 2nd reflection could be a different story. The 2nd reflection arises from the first reflection bouncing off the incorrectly terminated sending end and by the time it reaches the receiving end it may be enough to turn a digital 1/0 signal into a digital 0/1 symbol i.e. it causes inter-symbol interference.
So, for digital transmissions, longer cables can make the 2nd reflection trespass the timeslot of "later" bit and shorter cables have less of a problem. On the other hand, longer cables are likely to attenuate a reflection more so it's a little difficult to be general about this problem.
If in doubt, as a first measure, ensure the receiving end is accurately terminated. I hardly ever feed out to a coax or twinax via a line termination impedance so this should be OK. If you can't fix the receiving end then hope that the reflections are not significant enough to cause inter-symbol corruptions. For a CW signal I don't think you'll see total blackouts with a 100 ohm line being terminated in 50 ohms.