I am trying to understand how an ISP can write machine code into a microcontroller's memory.
In the case of AVR microcontrollers (such as the ATMega), you can use a special in-circuit programmer that will "engage" the target microcontroller over an SPI bus. (I am getting this information from this document)
What I'm wondering is this: if I had a fresh AVR with no prior programs written to it, how could it possibly "listen" to the SPI bus? Doesn't there need to be a program running?
A fresh AVR, while it has no code programmed into the user space, doesn't mean it has no capability.
You'll notice that the RESET line is used as part of the ISP protocol. While the RESET line obviously does what you'd expect for most of the chip, it also enables the mechanism that handles the ISP. It also clearly delineates 'normal' operation from ISP operation, you are in one, or the other. You'll also notice there's a handshake of 'magic bytes' for the programmer to establish that this mechanism is working.
Whether this extra mechanism does all the work to handle protocol and control the memory busses, or whether it just flips an address line to allow the main CPU to run ISP code from a ROM (sounds more sensible?), doesn't really matter from the programmer point of view.
The equally important question, is how do I reprogram my AVR when my user code hangs the chip (shortly after reset)...Nice to have a feature that works no matter what is or is not programmed into it.
As already mentioned, it is either a case of pure logic, or some logic and code in a rom used to implement this feature. Simple enough for pure logic.
Just because a pin is labelled reset does not mean the entire design has to be reset by that logic. Look at JTAG for example (on chips that have it), tends to have its own separate reset from the primary chip reset and is or may be completely separate reset domain from the primary chip reset. Reset is just another input that is used however the designers want, no automatic reason that every bit of logic has to respond to it.
There is obviously logic and perhaps some code that is not in that reset domain (is essentially in the power on reset domain, and/or inverted reset, when you release the reset pin this logic itself may go into reset and when you assert the reset pin this logic may get released).
