Well I finally decided to get familiar with PICs and got myself a PIC18F4680 and a clone of PICkit 2. I made a simple hello world circuit as shown here:
So basically I have an LED from port A pin 0, a crystal oscillator, one of those all in one units in a can (am I using it correctly?), two 100 nF capacitors and the MCLR pin pulled high (and connected to the programmer when I'm using it).
On the programmer, I have Vpp/MCLR, Vdd, Vss, ICSPDAT/PD, ICSPCLK/PGC and AUX pins. After some reading, I found out that the AUX should be connected to the PGM pin on the PIC. I connected all pins of the programmer connected to the respective pins on the PIC.
I've tried using the PIC with the external oscillator at 40 MHz and with internal oscillator at 1 MHz and it made no difference to the main problem. I think that the oscillators are fine because I can see a 250 kHz output on pin 14 when I'm using the 1 MHz internal oscillator and some strange reading when I'm using the 40 MHz external oscillator (which is to be expected since my multimeter can measure frequencies up to 5 MHz and I'm expecting 10 MHz output, according to the datasheet of the microcontroller).
The problem is that when I connect the programmed PIC to power, it won't do anything as far as I can see. Instead I need to connect the PICkit to it, open the Microchip's PICkit 2 application and after it verifies the chip, LED starts blinking as it's supposed to with correct frequency. When I disconnect the PICkit from the circuit or the computer, the flashing LED stops.