I am designing a IRIG-B AM modulated circuit. IRIG-B output signal will eventually be connected to another box (takes my IRIG-B signal as an output) through a wire (probably more than 4 feet). Essentially, the original signal gets created in the FPGA, then goes through a DAC (AD5761) and the output of the DAC goes through a buffer/difference amp (AD830) and the AD830 output goes to a SMA connector. The question is about the SMA ground pin. A senior engineer told me not to connect the SMA return conductor (outer conductor) to the digital ground of the PCB. Essentially, I am floating it so that I get the reference from the other box. Essentially, my output IRIG-B signal will reference the ground my box will eventually connect to. The senior engineer tells my that this is a good practice when we connect boxes through long wires (he says this could help with ground loops and EMI issues). Does it make sense? Is it always a good practice to float the SMA ground/return pins?
If you are using direct drive, you definitely need to connect the coax shield (and therefore the SMA "ground") to the ground of the AD830, and at the other end to the ground of the receiver circuit (or at least the return of an optocoupler or input transformer). Otherwise there is not a return path for your single-ended signal.
Whether your digital ground is connected to the enclosures is more of a topic for discussion. In some cases you may need to avoid ground loops, and float your circuit relative to the chassis, and in other cases you won't. Have a look at this answer for more details Should I connect a cable's shield to the ground layer of board when it's isolated from earth?
This is why BNC connectors come in metal and insulated version. However, SMA connectors generally don't come in an insulated version, so would generally be mounted directly to the case (though this could be on an insulated panel).
If there is a very strong reason to provide isolation of digital ground from chassis ground, you could use an output transformer on your transmitter, with the driver side of the transformer reference to digital ground, and the cable side connected to chassis ground.
This use of a transformer for isolation is often seen on digital comms interfaces, with ethernet being the most common example. See this answer for more details Why Are Ethernet/RJ45 Sockets Magnetically Coupled?
I had a nagging memory in the back of my head about grounding standards for IRIG-B, having used such systems in the distant past. The following doc IRIG-B Time Code Accuracy and Connection Requirements with comments on IED and system design considerations provides details of normal practice for IRIG-B systems.
Time Code Grounding Considerations for IED and System Design
From time to time, there is a discussion about how and where (and if) grounding of the time-code signal lines is required. IED designers can be tempted to use a non-isolated input in their device to save a little money. Best engineering practice generally requires any signal line to be grounded (earthed) at some point. For most analog signals, including time-code signals, this is normally the signal source.
Since ground loops are to be avoided, it is important to ground each signal at one point only. This must be the source if there is the possibility to have multiple loads attached to a given source. Therefore, time-code inputs in such a system must provide galvanic isolation.
There is also the system cost issue. Floating time-code outputs can be built, but require (costly) floating power supplies, whereas an isolated input requires no power supply. Compare a simple system having four IEDs driven by a clock: system A has one output, driving four optically-isolated IED inputs in parallel; and system B has a clock with four isolated outputs, each driving a single, grounded IED input. Clearly system A will have a lower equipment cost, since system B requires (in addition to optical isolators) floating power supplies for each independent output.
For these reasons, it has become best industry practice to ground time-code outputs from clocks, and use galvanic isolation of time code inputs to IEDs"
This is therefore pretty clear that your SMA output should be chassis ground referenced (with or without isolation transformer from your driver and logic ground), and that all inputs should be opto-coupled.
However, you can see this isn't 100% followed, because the same company also sell mid-line isolation transformers! 600 Ohm IRIG-B Isolation Transformer
The outer shield of a coax (which is what the SMA interfaces to) should be connected to the chassis of the boxes - that is, grounded - on each end. The outer shell of an SMA is usually connected to chassis. So if you're using standard SMA connectors & cables, this should do it. You would have to go through extra effort to not ground the SMA connector or the coax shield with this configuration.
At the source end (where the IRIG-B signal is generated), there should be a connection, or multiple connections, between the digital ground (return) on your PCB and the chassis.
In answer to your last question, it is NOT good practice to float the SMA ground/return pins. From my experience, this is the exception, not the rule.
A grounding diagram of your system would help me & others make a more definitive recommendation.