The same way we calculate airgap for any other kind of inductor.
Terminology:
- Transformer: a multi-winding magnetic component with very high magnetizing inductance, and generally a high coupling factor, so that transformed (instantaneous, induced) current dominates over magnetizing current; the energy storage during a cycle is negligible compared to the power transformed.
- Coupled inductor: a multi-winding magnetic component, intended to store and release a significant amount of energy over one or several cycles, generally through both windings.
This is far from a rigorous definition of course; we could have a component with some windings closely coupled to each other, but loosely to others, with high and low magnetizing or leakage inductances all around. The point is more to give
This terminology is not in widespread use. In commerce, "transformer" and "coupled inductor" have no specific meaning and are loosely interchangeable. You often see "flyback transformer" (a component intended to store energy), whereas a "common mode choke" is not intended to store energy. (A CMC is better described as a transformer; the choking action can be understood as a 1:1 current transformer, enforcing equal and opposite currents in the lines.) We might also call a two-winding component, where the secondary is very small (for purposes of auxiliary power output or voltage sensing), a choke or inductor, because the main (primary) winding is of, well, primary importance.
Language is a very squishy thing, so I add these details to be specific.
A "flyback transformer" is therefore better described as a "coupled inductor", and it also happens to need a high coupling factor. The fact that it needs two windings, is immaterial to the magnetic design -- the core sees the same total field whether it's one or many windings doing the work!
So, we can reduce this problem to that of inductor design. We do need to reserve double the winding area, as we need two windings which carry current alternately.
The basics of inductor design are:
- Pick an initial core. Doesn't have to be a good guess, but guesses will improve with experience. An ETD29 core is good for ballpark 100-200W at 100kHz, for example. Power scales with volume.
- Pick a flux density. Probably this will be from the core loss curves (look up the core material datasheet), or you can take typical values, for example in the ETD 29/16/10 | TDK datasheet they suggest N87 material at 200mT and 100kHz, and 2.8W is a reasonable amount for a core that size. (Permissible power dissipation depends on available cooling, and desired efficiency; if you need very high efficiency, the required core size can be dramatically larger than usual!)
- Pick an effective permeability, typically in the 20-100 range. The rationale for this is too lengthy/complicated to get into here, but in short, this figure derives from the fact that copper has the resistivity that it does. (Consequently, if you're using a higher resistance material like aluminum, or the part must operate at high temperatures (most metals have a strong positive tempco), aim for a higher permeability. Conversely, if we had significantly better conductors than copper, we could get by with very low permeability; as it happens, superconducting magnets are quite effective with ludicrous numbers of turns, of very fine wire, though they're also air-core by necessity as they can operate far above the saturation level of any ferromagnet.)
- Notice the winding area (\$A_N\$, or often \$W_A\$ or etc.) and inductivity (\$A_L\$). The latter determines how many turns you need, and the former determines what wire size can fit those turns onto the bobbin. Don't forget to include a winding factor: at the very least, round wire has a maximum packing density, but it's common to be able to use merely 70%, or 50% or even less, of the total winding area. Triple-insulated wire may also help out in smaller transformers.
- If you find the winding would dissipate too much power, or after checking flux density at design parameters you find the core losses would be too high, then restart from the top with a larger core, and repeat. Or conversely if it's very low and you wish to save some cost and space.
This should give you a basic explanation, albeit stopping short of the full equations written out.
And yes, the iterative process is something of a PITA; fortunately there are design tools available on most manufacturers' sites, and core data can be compiled into a spreadsheet where the calculations can be ran instantly in parallel, finding the best core among them for a given application.
There also exists an analytical solution, to find the maximum-Q point of a given core geometry and wire resistance, assuming the core material can be reasonably described by a generalized Steinmetz loss relation; see Fundamentals of Power Electronics, Erickson and Maksimovic (2001), §15.4.2 for example. I have found this approach to give a reasonable starting point, though it consistently overestimates Q factor by 20-80%.
As for DCM vs. CCM, the AC losses only depend on the AC component of course, so core loss and winding ACR are weighted lower for CCM than DCM. DC winding resistance is present in both cases, of course. From perspective of the magnetics design: CCM permits higher power levels from a given core size, or higher loss materials to be used (such as powdered iron). For example, at 100kHz, a 40mm toroid in #52 powdered iron might be good for 30VA of reactive power, or about 30W at 100% ripple fraction (BCM); at 10% ripple (CCM), you get the same AC component but 300W of DC output!
I'm afraid I don't have a feel, offhand, for what DCM vs. CCM would do to the air gap length in a fully optimized (all variables solved for) design. It seems like a question sideways to the correct design route; not that it would be a meaningless question, but that it's not the most important or meaningful one.
The choice of DCM/CCM is typically defined more by the controller, or overall economics of the design, than the magnetics. (Of note: peak current mode controllers require a high ripple fraction, indeed ≥ 100% (DCM) without slope compensation.)