I feel bad for saying this considering the amount of work that went into this board, but the layout of the DC-DC is terrible.
As Jonathan says you should be able to salvage this by putting the HV switching converter on a separate board. I'm going to explain how to build it.
First look at the pictures in this answer.
Get a bit of blank copperclad which will be your ground plane. It should be roughly the same size as your current DC-DC converter so you can put it in place later, it looks like 3x6cm or something like that.
Then take a bit of perfboard, say 2x5 cm. Put this on top of the copperclad, upside down so you see the copper pads. Use 4 bits of resistor tails in the corners to solder it to the ground plane and keep it in place.
Now you can assemble your circuit on the perfboard. The component leads won't be able to go through because there is grounded copperclad on the other side, but that's not a problem. Make the important connections as short as possible. That's mostly the current path through the input cap, inductor, FET, diode, and output cap. Optimum layout has the input cap, output cap, and FET ground pins connecting to ground at the same point.
The important thing here is that the copperclad gives you a very low inductance ground plane and you can solder the ground pins of all components on it directly. It will also act as a shield and reduce EMI radiation.
Now about the component choices... let's run the equations for your boost converter.
Vin=12V / Vout=180V / Iout=20mA
Duty cycle D = 0.93 (which is way too high for a boost, a transformer-based converter would be better here but let's go on...)
Frequency is hard to read on MC34063 datasheet graph but let's say F=50kHz.
T = 1/F = 20µs
Toff = (1-D)T = 1.33µs
Ton = DT = 18.66 µs
(Toff is a problem as I'm not sure the wimpy gate driver will be able to do a full on/off cycle in 1.3µs but let's go on...)
Inductor ripple current = Ton * Vin/L = 0.68A YIKES!!!!
At 180V/20mA out, you should have around 300mA average inductor current but since the ripple current is more than 2x that, something will have to give somewhere...
Now let's check the caps:
Ripple current rating of 2.2µF/400V capacitor on the output = 34 mA !... This cap is useless anyway, as it has an ESR of 40 ohms (random datasheet from Mouser). So this cap will shortly die, and it won't smooth the output voltage.
Problem with the high ESR of this output cap is that it will result in huge voltage spikes on the output when the FET switches. Plus the layout inductance which doesn't help. This could be what sends your microcontroller in a coma. With 40 ohms ESR when the FET turns off and the inductor dumps its current into the output cap you're looking at a 20-30 volts spike with a rise time equal to the FET turn-off time, which is rather fast...
Fix: use a low-ESR cap with a proper ripple current rating on the output. This is going to be a bit hard to find, unless you use ceramic or film, but at 400V these will be huge and therefore inductive. (This is also why a transformer-based converter would work a lot better, output current ripple is much lower).
The input cap (on 12V) also needs to be able to provide 0.68A ripple at 50kHz so it needs to be a beefy low-ESR one. Check the datasheet. Add ceramic caps in parallel.
I checked this dude who sells a kit with the exact same schematic, and it doesn't seem to bother him at all that the output cap runs at more than 20x its rated maximum ripple current. I wonder how long it will last...
In light of this I would rather advise you get a transformer-based converter, maybe this one, I never had it in hand so I can't vouch for it though. It's available on ebay for about $10.