I have a differential analog signal traveling over 2 coaxial wires. I need to convert to single ended. When should I use a BALUN ? When are other solutions such as OPAMPS more appropriate ?
Use the appropriate one for the frequency range.
Anything less than 100 kHz, use an opamp. Anything over 50 MHz, use a transformer balun. Between those frequencies, depending on your signal level, or distortion requirements, you'll need to choose your component carefully, paying a bit more for a faster opamp, or testing your magnetics critically.
The optimal solution depends on the signal frequency and strength, and in some cases also on the available technology. At high frequencies - I would say >n*100MHz - balun is used, because no transformers or amplifiers are available, below that frequency transformers are the first choice, while at/near DC feedback based circuit techniques: opamps, or even just a source coupled transistor pair where only one of the outputs are used.
If you happen to work on an integrated solution, then transistor based solutions will work even up to few 100GHz depending on the actual technology being used. Similarly you can use RF transistors on a PCB for more than 100MHz, but the preferred choice between this and a balun involves considerations of the cost of the amplifier, the PCB, and the assembly. A passive structure always require less care once it is designed.
RF transistors and transformers might be limited by their rated input power. There is a limit for dielectric breakdown and electromigration for planar PCB baluns as wells, but it is way more relaxed.