You are trying to use an N-channel MOSFET for a high-side switch, and you are not turning it on, because you are not driving the gate correctly.
For the MOSFET to turn on, the gate should be a certain voltage (VGS)about MOSFET's source. If the source itself is at some positive voltage, then the VGS has to be on top of that voltage. Imagine that you want to have +12V at the positive terminal of the pump, and you want to drive the gate with VGS=5V. Then the gate voltage with respect to ground would have to be +17V. You don't have a high enough supply rail in your circuit for that.
How to explain the power dissipation which was observed? When you try to turn the MOSFET on, the voltage at the positive terminal of the pump was somewhere between 0V and +5V. There is still enough voltage to turn on the MOSFET partially. As a result, there is still some current flowing though it, and there is a voltage drop across it. It can't turn on fully, and it can't turn off fully.
Options to fix this:
- Use the same N-channel MOSFET as a low-side switch.
- Use a P-channel MOSFET, if you need a high-side switch for some reason.