It can be very difficult to calculate EMI filter values, since EMI is more dependent on the circuit's physical layout than the topology itself. The parasitic elements related to component spacing, shielding, pickup, coupling, etc. will outnumber the amount of actual parts. Often a more empirical approach is needed.
The EMI solution cannot be analytically 'proven' as good either. You need to take measurements with a proper spectrum analyzer and LISN (line impedance stabilization network) under multiple conditions and prove empirically that the complete product meets emission standards.
Your converter will generate EMI at it's switching frequency and at harmonic multiples of that frequency. If you have other switching elements in the circuit, you may also see sum and difference frequencies. The ultrafast rectifier diodes will generate high-frequency noise (usually up into the megahertz) due to their speed.
Obviously, the EMI inductor has to carry the line current. In the common-mode configuration, the input current will cancel out so you don't have to worry about it saturating, but the wire should be thick enough to carry the current without \$ I^2R \$ losses.
As Russell was saying, a proper common-mode filter will combine a common-mode inductor (phase dots on the same ends) with some Y-capacitors (a cap from each winding to protective earth). The capacitors used for this function MUST be safety-approved for Y applications. Your design should include at least one set of Y-capacitors.
The two across-the-lines capacitors (Cin1 and Cin2) are referred to as X-capacitors (since they 'cross' the mains). These act more on differential-mode noise, along with any differential-mode inductors.
As a starting point, you should ensure that your filter has lots of attenuation at the main switching frequency of the converter, since this is often the strongest EMI source in the power supply.
For a common-mode inductor, this is generally achieved by having the inductance as high as possible, to ensure the highest inductive reactance at the switching frequency. This usually implies high-permeability ferrite material, to achieve the inductance without needing many turns. Toroidal cores are often used as they easily allow the line-side and neutral-side windings to symmetrically fit on the core, usually with an insulating spacer between the windings.
X- and Y-capacitors are a little easier - their datasheets will have characteristic curves that show their attenuation as a function of frequency.
Once you settle on your capacitors and inductors, it's time to start measuring, tweaking, remeasuring, retweaking...