Is it unrealistic to have a 125 MHz sampling rate with a cutoff frequency of 5 Hz? I downloaded a few filter programs and used their methods and it seems that even with thousands of taps it still looks pretty bad. Is there a better way to implement this digitally or should I just stick with an analog implementation?
The basic formula for a simple filter relies on the ratio of time between samples and the time constant of the filter i.e. T\$_S\$/CR.
At 125MHz Ts is 0.000000008 and CR = 0.03 for a 5Hz cut-off.
Quite a few digital systems are going to produce errors unless you are working with decent floating point numbers. This is a simple digital IIR filter with a nod to the CR time of an analogue low pass RC filter: -
It's dead easy to implement in a spreadsheet and you can apply integer math (or whatever you are using) to see how it performs with the vast difference in sampling frequency and target cut-off frequency.
Analog will be simpler (or maybe not - a circuit that works correctly from 5Hz to 75MHz might be difficult to implement,) though you can also use a digital implementation of an IIR filter.
FIR filters (which is what you tried to implement, judging by use fo the term "taps") will require a lot of taps for a 5Hz (assuming high pass, else why would you need 125MHz) filter at 125MHz sampling rate.
An IIR filter can do nearly the same job with just a few adds and multiplies. Look around for filter design software and algorithms for IIR filters.
It might also be fun (read as "challenging") to build an active filter with a cutoff of 5Hz that can still pass frequencies up to 75MHz.
I think your best bet really is a digital IIR filter - provided you can guarantee that the signal content below 5Hz won't push the ADC input past its limits (for example, a 2Hz signal at 5V peak to peak when your ADC can only take 3V peak to peak.)
If one has a stream of 125M samples/sec, and one wants a stream with that same number of samples that only contains components that are 5Hz or below, the best way to accomplish that is to repeatedly perform a combined low-pass filter/downsample operation. A 100 sample/second stream is going to contain just as much information about signals 5Hz and below as would a 125M sample/second stream, but a low-pass filter with a cutoff at at 1/20 of the sample rate will require many fewer taps than one with a cutoff at 1/25,000,000 of the sample rate.
If one needs a high-pass filter rather than a low-pass filter, one can use symmetric low-pass filters in all the down-sampling operations, and then reconstitute the higher-speed signals from the lower-speed ones (again using symmetric filters) so that one ends up with a 125M samples/sec signal whose 5Hz content will be delayed by a fixed amount of time relative to the original. Delay the original signal by that same duration and subtract the reconstituted signal from it, and the result will be a signal which contains all of the high-frequency content of the original but none of the low-frequency content.
Note, btw, that at lower sample rates, the amount of work per second required to implement a filter with N taps will go down. If the first stage downsamples by 50%, the next stage will only have to do half as much work as if it was operating on the full-speed original signal. Later stages could afford to use more taps, and could thus afford to downsample by larger amounts.