I remember reading long ago that I don't need filter caps for a watch (32768 Hz) crystal. Is this true? Or will I need to put some capacitors to ground on either end of the crystal? I will be using the crystal as the primary clock source.
Atmel says Crystals specifying load capacitance (CL) higher than 6 pF, require external capacitors applied as described in Figure 9-2 on page 38.
6pF Xtals do exist but will not be as stable unless your layout includes a ground plane under the XTAL to create this added capacitance, the tolerance of the crystal cannot be met. But 50 ~100 ppm error may be close enough for you.
With no ground plane under the tracks or Xtal there is much less stray capacitance.
A Saturn PCB design simulation (free) can prove a layout capacitance if you really want to eliminate the caps.
0.127mm 1oz tracks over 0.127 thick FR4 is 0.92 pF/mm so 6.5mm track length would give 6pF on each track. Cin on chip is <1pF.