I implemented an RMS function for an oscilloscope program not too long ago, so I have a little practical experience with how the algorithms work.
DKNguyen gave the basic mathematical definition in another answer.
What's missing is that naively applying it to a signal will result in inaccurate values.
If you can capture exact multiples of the signal frequency (number of cycles or half cycles,) then simply applying the RMS definition will get you an accurate number.
It's kind of tricky to do that with arbitrary signals with arbitrary frequencies. How do you tell for sure when you've got a full cycle of some signal with a mixture of frequencies?
To get an accurate value for arbitrary signals, you need to be certain that you get enough cycles of the input signal. Your sampled data needs to include about 8 cycles of the signal in order to get the error low enough to be ignored. I can't find it now, but I had a reference that explained that with more than 8 cycles the error would be less than 1%.
The simplest way to guarantee that minimum of 8 cycles is to pick a lower frequency limit and calculate the RMS function on a block of samples at least 8 times longer than one cycle of your lower limit.
As far as I know, multimeters that calculate RMS values all have a stated frequency range for given accuracy.
Take the Fluke 179 True RMS multimeter as an example.
Here are the specifications for the AC accuracy:
Note that there's a lower frequency limit. As you go below that frequency, the RMS values will be increasingly wrong.
That ties in with the minimum number of cycles needed to get accurate results.
I would expect the meter to integrate the RMS over at least 167 milliseconds.
Setting a lower frequency limit for the input signal and implementing a minimum integration time is simpler than trying to synchronize to the signal and capture some specific number of cycles.
The RMS function I implemented in my oscilloscope software just ignores the problem. It calculates the RMS value directly from the definition. The user (me) is expected to use a sweep speed that captures enough cycles of the displayed signal. It will happily display RMS values to 3 decimal places that are in fact more than 100 percent wrong when the sweep speed is too low.