I am sampling a pure sine wave for 1mS at 200nS conversion rate over a 12 bit range, getting 5000 samples. I need to know how accurately I can determine the RMS value of the sine wave. More generally, for a full scale resolution of M bits over N samples, how accurate can I get the RMS amplitude? [I should add - the frequency is 125kHz ie 8uS period]
In order to fully reconstruct the sine wave and avoid aliasing problems, your sampling rate should be at leas the Nyquist rate which is twice as the sine wave's frequency. Assuming this condition holds, you can fully reconstruct the sine wave(up to ADC's resolution) and calculate the RMS. The ADC resolution along with your scaling factor will determine the accuracy of the measurement. If we assume that the resolution is X mV, the RMS error will be X/sqrt(2) mv.
If your only source of error is quantization noise, which is uniformly distributed and uncorrelated among samples, then your RMS measurement will be averaging it out, increasing the resolution of your measurement relative to the "raw" resolution of your ADC. This is a form of dithering.
The effective error will decrease proportionally to the square root of the number of samples. This means that if you have a peak-to-peak error of 1/4096 = 2.44e-4 (relative to full scale), with 5000 samples this will be reduced by a factor of about 70.7, giving an effective error of 3.45e-6.
Note that if your sample rate is frequency-locked to the signal, then the assumption of uncorrelated error may not hold.