I have an output transformer, which is a clone from an original Fender 5e3 transformer, I would like to know how it's behavior in typical frequencies. How can I perform such test using instruments like osciloscope and frequency generator, is It even possible?
You can use a signal generator and oscilloscope. Set it up like this: -
Connect the scope's 2 channels to A and B with the scope inputs grounded at the commoned point of the transformer and signal generator.
One channel (A) of the scope verifies that what you are inputting stays constant and you note down the value (p-p) on the scope channels connected to (B)
This test only works at small input levels but if you are trying to compare two transformers it'll work fine.
This is probably better as a comment as I'm speculating and have not done so myself, but it was too long and I invite comments because I want to learn as well.
Output transformer properties depends on the power it is transferring as well due to saturation of the magnetic field in between the windings, I would assume that the output transformer was chosen to avoid driving it in saturation for the amp specs, but even then, the relatively high currents in the output stage would potentially drive it into a non-linear regime as the transformer gets off the ground magnetically speaking. Even if it doesnt get near saturation, the heat dissipated can change the frequency response as the amp warms up. Additionally, guitar amp designers like to chase "tone" and it may be intentionally driven near saturation to exploit the skewed frequency response.
If your generator is good You could test the baseline frequency response by pumping the generator through the transformer and sweeping the frequency in the audio range, if you don't have a spectrum analyzer you can do this point by point and measuring the signal amplitude on the output stage as you modulate the frequency.
If you have both transformers on hand I would just suggest to pump your frequency generator through the amp to see the relative difference between the two transformers, however since tubes are distinctly non-linear in their frequency response (and used for music for their characteristic "tone" due to this response) it wouldn't be that useful if you just had the one transformer on hand
Otherwise, building a very clean amp setup with silicon with a linear frequency response (or close to it) outputting a similar power to the tube amp to drive the transformer would be the best practice I would assume. However if you don't have an amplifier on hand you will have to spend time and money building one.
What I did to test some valve output transformers was rough and ready .My audio sig gen had been modded earlier with an emitter follower to test speakers .I back to back connected my two transformers so things were 8 ohm to 5Kohm and back to 8 ohm .I then run the low output impedance gen into the composite 1:1 audio transformer .I connected a 8R2 9 Watt resistor aggregate made from 9 series parallel 8R2 carbon film resistors .I was not sure about the inductance of my ancient junk box wirewound resistors .Anyway I put one scope probe on the input and one on the output and swept the gen from 20Hz to 20 KHz .