I am developing an high current dc motor driver, and I'm looking for mosfets that can dissipate enough power without heating too much thus without using heatsink. I saw this project : http://navrobot.blogspot.it/2014/12/high-current-dc-motor-driver.html but I made some calculations and I found out that the maximum current that can flow without heatsink is around 3A. Do you know any mosfet that can deliver enough current (about 20A) for my project?
You have made the calculations so you should know what Rdson is required. For 12V supply a 25V VGS and 30V VDS rating will be sufficient (higher voltage FETs generally have higher RDSON and so should be avoided). To find suitable FETs, use the parametric search function on your vendor's website.
Alternatively you could wire two or more weaker FETs in parallel to reduce total RDSON and also increase dissipation capability. Two packages have twice the surface area so they can dissipate (almost) twice as much power, and since each is carrying only half the total current it dissipates 1/4 the power of a single (same spec) FET. Of course they will also take up more space, but may still be more compact than a single FET with heatsink.
However low RDSON comes with a price - high Gate charge. The circuit in your link will probably not be suitable because it has very weak Gate drivers. If you are using PWM the low-side FETs will need to be switched on and off rapidly to avoid excessive power dissipation during the transition period (when the FET is carrying both high voltage and high current at the same time). You need high current drivers that can charge and discharge the Gates quickly.
You can make a high current Gate driver with discrete transistors, or just use an IC such the TC1428 (which has both inverting and non-inverting drivers, so you only need two ICs to drive all the FETs).
You can do this with no heatsinks .Keep the PWM freq as low as acceptable audible motor noise will permit .Motors are different but but you should be at 20KHz or less .Low frequency operation of low voltage mosfets will mean that the switching losses are well below the conduction losses and for your ballpark calculation can be neglected .I am assuming that your battery voltage is low .Your P chan fets allow simple circuit but they do have more on resistance .The reason for this is electron mobility Vs hole mobility .It may be better to use an all N chan H bridge and a high side driver .Say if you google some 10milliohm devices that are in say a D squared package which are easy to hand solder .Mosfets share well when in parallel so think about 2 in parallel which means 8 devices .Now each device could get 10 amp and would waste 1 watt . 1 watt on a D squared pack on a well designed PCB with 2 oz copper or better with lots of Vias will be fine for operation at normal ambient temps .You could employ a hall effect device to allow the power wasting current sense resistors to be removed .