Although you didnot mention the current rating of your motor and the available supply voltage I see 3 Solutions to your problem :
1-First to drive a Permenant magnet DC motor bidirectionally using an H-Bridge. you can use all N channel (this is the common practice) but you have to provide something called high side gate drive , this can be using pulse transformer , or gate driver IC , etc ... please note that even then low side mosfet need 10-15v on its gate to work properly unless it is a special type called (logic level or trech) mosfet which work at 4.5v to provide low RDS. Since your motor is 4v , i assume your battery is close to 4 maybe 6v ??
2-Another solution is to use a single mosfet in low side configuration , and use a double throw relay to reverse the motor wires . but this requires the motor to stop completey before changing direction.
3-The option most prefered for small scale applications is to buy an IC that contains the mosfets and drivers built in , inaddtion to current and thermal protection , for example : L298 L6201 DRV8816 TLE 5206 LMD18245.