That would depend on the specific modulator, but RF signal output is just basically the same composite video it gets but modulated to RF.

So the RF output will be NTSC-M if the baseband CVBS that is sent to modulator is also NTSC-M compliant, and that the modulator modulates it in NTSC-M compliant way.

Some sources can/may output non-broadcast compliant CVBS baseband signals, because the output is meant to be connected to a TV or other equipment, and not meant to be transmitted over RF in a compliant fashion. The modulator can and will most likely limit the bandwidth to fit the signal to the RF channel, but it will still modulate whatever compliant or non-compliant signal it gets in.