Getting the power down on embedded processors is not trivial! And, on most ARM core processors, there are ways of turning off many individual parts of the chip. You can save even more power if you slow the clocks down. There are usually many ways to do this in an ARM core chip.
(Don't stop reading at the end of the processor vendor's specifications. Determine if the ARM core specifications provide more insight into how to control the internal workings of the chip.)
Permanent pull up resistors are never a good idea when power consumption is of a concern. Consider removing the permanent external pull up resistor. Then consider using a pin with a programmable pull up resistor which you only engage when testing the external switch.
In this case, you are using the external signal to wake up the processor. Consider using an internal timer to wake up the processor and check the state of the switch. Many vendors have multiple power level options where periodically waking up and checking the status of a switch consumes a trivial amount of power.
Many power consumption problems can be solved through proper hardware design. But clever software will be the final battle ground for reducing the average current down through the last few mAmps down into the uAmps and even the nAmmps range.