Too long to read all that stuff so, here's the short story
From comments under the question, Trevor says this: -
we think the caps are correct. We have 2x 15pf to ground. The crystal we're using is this one
But, the part linked is a 6 pF device (that requires 2 x 12 pF tuning): -
So, you are probably using the incorrect loading capacitance and, below (the long-story) is how that might change things...
A bit of background stuff first
The crystal model below was made by looking at several off-the-shelf offerings of 10 MHz crystals and then forming an "average" equivalent circuit. Either side of the crystal are CL1 and CL2 (the loading capacitors). In series with the driven end (V1) of the circuit, is R1. All these components ensure that the crystal "sings" at the frequency written on the tin: -
The model values cause a series resonance of precisely 10,000,000.1403 Hz. But, a crystal can never be used at that exact frequency; it has to operate slightly "inductively" as per this diagram: -
Let's move on to frequency errors by looking at the AC response of the full simulation.
Crystal frequency errors due to changes in capacitive loading
The plot shows some typical frequency changes when capacitive loading is altered in 5 pF steps: -
Important here is the frequency at which the phase change between Vin (the inverting logic gate's output) to Vout (the signal fed back to the inverting gate's input) is 180°. Because the gate is an inverter, you get oscillation when the phase change is precisely 180°.
With 2 x 20 pF loading, the oscillation frequency is 10.00137 MHz. If the loading changed to 2 x 15 pF, the oscillation frequency becomes 10.00165 MHz. That's a change of 280 Hz in 10 MHz and, is equivalent to 28 ppm.
Do you see the issue here?
Slight changes also occur due to the value of the driving resistor changing
It's a smaller effect than that seen by changes to the loading capacitor but, nevertheless, it's an effect that could be improved upon. Results above are with a 2 x 20 pF loading.
Gate propagation delays can cause changes
No matter how good the crystal is, or how carefully you have chosen the component values around it, if the inverter gate is a poor performer, you will get oscillation frequency errors and possibly excessive frequency drift. Consider the 74AC04 inverter: -
The propagation delay figures quoted for rising and falling edges are typically around 5 ns but could be as high as 10 ns. This means an overall output delay time of 10 ns and, considering that we are looking at a 10 MHz oscillator, 10 ns is equivalent to adding 36° of phase shift. This is alleviated by the type of circuit used; we are operating the gate in a semi-linear fashion and, deep saturation of the output MOSFETs would not happen but, something like 20° of equivalent phase shift might be expected.
This means that the loaded crystal circuit only has to produce a phase displacement of 160° in order to create oscillation: -
So, basically, if the gate that is used to turn the crystal into an oscillator is a little flaky and drifts, you get more errors. OK, this is less of a problem for a 32,768 kHz crystal than a 10 MHz crystal because gate delays represent a proportionally smaller error.
However, you still need to check that you aren't using something really crappy because, the error could be significant.
Pictures (and some text) taken from my crappy website.
Summary
- Get the crystal loading capacitance right - use good capacitors and ensure you understand what the gate's input capacitance and other parasitic capacitors are.
- Make sure you are using the correct value series resistor. For a 32,768 kHz crystal this is exceptionally vital (not just because it might give an error but because you could easily damage the crystal with too much power).
- Make sure you have a decent gate for the oscillator.
Can I ask, what types of crystals do people use when building a normal
time clock?
And
So what should we be using? A 5ppm crystal or a 20ppm and manually
adjust each one?
It sounds to me like you need to observe the above and note that you can tweak a crystal into more initial accuracy.
Reminder about the loading capacitance discrepancy
The mouser link to the crystal appears to say that the loading is 6 pF so ideally, that is formed by 2 x 12 pF capacitors and not 2 x 15 pF capacitors. Then, you should probably assume the gate input has 2 pF of capacitance and, that there may be 0.5 pF of parasitic capacitance across the device due to the PCB and its tracking.
All in all, it's looks like you may be over egging the loading capacitance and, quite possibly, you should be aiming for more like 2 x 10 pF and not 2 x 15 pF.
Double check your drive power too because with loading at 2 x 15 pF you might be putting too much power into your crystal (the drive level is quoted in the data is 1 μW maximum.
As an afterthought, if the device is predictably running different to true time, then, you could make a compensation in software. But, you need to be very certain about this so that it's not some long-time-based cyclic drift.
