If you have no technical requirement that allows you to choose between two design options, then you will want to choose the option with the lowest cost.
Choosing between a linear and a switching power supply, the linear option will typically be the lower cost one. But if the power dissipation gets too great, the heat sink required for the linear supply could conceivably increase the cost enough to make the switching option preferable.
How do we decide or calculate, that for a particular application, SMPS or a Linear Power Supply would be better.
You'd have to consider the requirements of your application first, in order to determine what "better" means for a particular design. It might mean lowest cost. It might mean most efficient. It might mean lowest voltage ripple. It might mean smallest board area or least weight. Most likely it will be a balance between several parameters.
In some applications we might even choose to use both: A SMPS pre-regulator to achieve reasonable efficiency with a linear regulator to reach the final output voltage with low ripple.
How do we decide or calculate the losses in both the system, for a particular application, ...
You'd have to design two candidate supplies, and then analyze the efficiency of each one.
i.e., if the core losses would be more in a Linear Power Supply, or the switching losses and the core loss of an SMPS would be more.
Unless the output voltage is very close to the output voltage (say \$V_{out}>0.9 V_{in}\$) you will almost certainly be able to design a more efficient switching supply.
