I've got a very simple circuit (as you can see from the diagram below) with an ATTiny45 and a piezo buzzer. Everything worked great as I was testing it out and had the ISP (an AVR ISP MkII) attached. The piezo played the little tune it was supposed to, every thing was fine. But when I disconnected the ISP from the ICSP header everything goes haywire. The speaker just kind of constantly stutters. It almost seems like make it's constantly resetting but I'm not sure why. I tried placing a resistor between VCC and RESET to make sure it's pulled high, but that has no effect. I've also checked everywhere and there's no shorts that shouldn't be there.
Any thoughts of what the AVR ISP has that my circuit doesn't and obviously needs?