In the BJT circuit the switching transistor should be PNP with a second NPN transistor to provide the drive current to it.
My circuit below is remarkably similar to the one Russel posted and it has been tested with an Arduino UNO. It switches the twelve volt supply pre-regulator.
The push button injects current into the NPN transistor which in turn injects current into the PNP transistor that powers up the Arduino. With the UNO the capacitor is required because it takes a long time for the UNO to start up and to set the ON_PIN high. The capacitor discharging through the 5k6 resistor keeps the base of the NPN supplied with current until the ON_PIN takes over.
The circuit on the left with the diodes allows further push button presses to be sensed so that you could signal other functions to the code or, as I do, use it as an abort.
I have used variants of this circuit with batteries to power IR and WiFi remotes using other MCUs, e.g. Adafruit ItsyBitsy. When the circuit is off the leakage current is under 1 micro amp making batteries last months between charges. Circuit values have to be adjusted for other supply voltages.
The above is a solution to your initial problem but, to answer some of your questions:
For the BJT circuit, it turns on when the button is pressed because current can flow into the base of the transistor from the supply which is at a higher voltage than the emitter of the transistor. However, it doesn't work when the CPU output D2 should take over because, to turn the transistor on the base has to be around 0.6 volts higher than the emitter. The output pin can never supply more than the Vin of the CPU but Vin is provided by the emitter of the transistor. This is a vicious circle and current can never flow into the base of the transistor because the base voltage never gets high enough. Add in the 0.6 volts drop in the diode forward voltage and the situation is made worse. That is why I use a PNP for the switching. If the emitter is connected to the supply then you just have to connect the base towards ground through a resistor to turn it on.
For the FET the answer is similar but, as others have said, the current needed by the gate of the FET is tiny, and the charge on the gate of the FET will hold it on until it leaks away. There is no mechanism in the circuit to actively discharge the gate when the D2 output is pulled low and that is why it doesn't switch off under D2 control.
You say, "Presumably this is due to the 100uF capacitor", exactly. The capacitor charges almost instantly from the push button and discharges at a rate controlled by the 5k6 resistor. With other boards (ItsyBitsy and Feather) I didn't need the capacitor but, with the UNO I did. The start up time that I saw was around 1.5 seconds.
Another feature of my circuit that I like is that, although it is started by a button here, it can cactually be triggered by any sensor that can provide enough current to turn on the base of the BC547, e.g. a door switch, a temperature or light sensor going above a threshold, etc.