I am making a project where i need to control the speed of a DC 12v motor (10A) and i am confused about which kind of transistor to use in this scenario. so i need your help to suggest me transistor which can handle 10A without heatsink and support PWM with arduino.
Use a N channel MOSFET as a low side switch. Since you have 12 V available, don't try to limit yourself to "logic level" FETs.
At 10 A, low Rdson will be very useful to avoid lots of dissipation. For example, with 10 mΩ Rdson, the FET will dissipate 1 W when on and the full 10 A is being drawn. The power dissipation is proportional with Rdson from there.
You will find a better set of tradeoffs available if you look at FETs intended for 12 V gate drive when fully on. All you need is a FET driver chip between the logic level output of the microcontroller and the FET gate to generate the gate voltage.
You want a MOSFET with less than 0.005Ω RDSon @4.5V VGS (eg. FDP8860). Current rating will probably be 80A+, but this is not as important as the Drain-Source resistance.
Power = I2*R. At 10A a FET with 0.003Ω RDSon would dissipate 10*10*0.003 = 0.3W.
A TO220 package has thermal resistance to ambient of 62°C/W. Dissipating 0.3W would raise the case temperature by ~62*0.3 = 19°C. If the ambient temperature was 25°C then the case temperature would be ~44°C.
You also need a 'flyback' diode across the motor, to recirculate current generated by the inductance of the motor windings during PWM 'off' time. Without this diode the motor voltage will spike until the magnetic energy finds somewhere to go, which will be into the MOSFET at its breakdown voltage. The diode should be a 'fast switching' rectifier rated for at least 3A, preferably Schottky type for low voltage drop (eg. 1N5821).
The circuit should look something like this:-
R1 ensures that the FET will be turned off if the Arduino PWM pin is a floating input (eg. while booting). C1 reduces voltage ripple on the power supply caused by the pulsing current drawn by the motor.