With battery positive on N and battery negative on P you have a reverse biased PN junction. That forms a wide depletion layer that is basically non conductive unless the breakdown voltage is exceeded.
The majority carrier electrons in the N material are attracted to the battery positive terminal but, there is only a limited supply of them and this produces a really tiny glitch of charge movement into the battery that is unsustainable.
It's a similar story for the majority carriers in the P material: -
Image from here.