I have a MOSFET (IRF840) hooked up as a DC motor switch. Source is connected to ground, drain is connected to the motor. When I set the Vgs to 5V, the Id is about 1,5A, but the Vds jumps to over 3V. Can someone tell me why the Vds gets so big?
Look at this in the data sheet (I've drawn a red line on at 1.5 amps): -
These are typical values and what they tell me is that if your Vgs voltage is more like 4.5 volts then you'll never drive the motor properly. At 5 volts it should be OK but probably at somewhere between 4.5 volts and 5 volts you might see 3 volts dropped from drain to source.
Remember these are typical numbers and for the particular device you may have you might need a little bit more than 5 volts to achieve what you want. You should also remember that this device isn't a low on-resistance type - even at 5 volts and 1.5 amps drain current, the volt drop from drain to source is a fraction over 1 volt and you will be dissipating over 1.5 watts of power.
Missing information as to whether the MOSFET can handle the current that the motor will take. But that said lets assume that the current is reasonable for the MOSFET in use. If the VDS is too high it means that the FET is not fully enhanced and will need to have a higher VGS than the 5V you are giving it.
The IRF840 FET that you are using has an RDSon spec 0.85 ohms when the VGS is at 10V. So even if you gave the VGS at this level the VDS of the device will be equal to about 0.85 * Imotor. (Imotor = the running current of the motor in Amps).
Depending on the voltage applied to the gate, the Mosfet has a different on-resistance (Rds_on). If you look in the datasheet of your IRF840 you can find that with a gate voltage of Vgs=10V the Rds_on is still 0.85 Ohms. So even with a higher gate voltage of 10V you would still have a Vds of 1.5A * 0.85 Ohms = 1.275V. Given that the applied gate voltage is only 5V, Rds_on will be larger. Thus the larger Vds.
If you want to reduce Vds further you need to increase your Vgs. The IRF840 can handle a maximum of Vgs=20V, so something like 15-18V would be best. You could create this voltage by using a boost circuit, charge-pump or find a Mosfet with lower Rds_on.