Somethings not quite right there.
I have the 200MHz SD8202 which has the same manual so I assume the procedures are the same. It has a 5V pk-pk compensation output. I'm guessing you have adjusted the vertical position to zoom in at the top of the waveform.
I just tried this and I get similar to your top picture but with a bit less noise (did you connect the ground clip too? Was it bandwidth limited to 20MHz like the second waveform?)
All I can think of trying right now is running the self calibration routine (remove probes, press "Utility" button, press "Function" (H1 button), choose "Adjust" with "M" knob, press "Self Cal" (H2 button)
Let this complete (couple of minutes) and try the probe compensation again. Let us know how it goes.
EDIT - What do you get with all the waveform on the screen? (e.g. 1V/div)
It's possible the front end is slightly different in your model (or conditions/part tolerances are different), and it's at the limit of the vertical amplifier as Zebonaut proposes.
I'd also try a divider on the compensation signal to make it so the full waveform is on the screen at 20mV/div, just in case it's a problem specific to that gain setting.
At 5V pk-pk, to bring it down to say, 100mV pk-pk you need a 50:1 divider. Something like 50k and 1k will get "close enough" (98mV pk-pk) Or use a potentiometer and just adjust visibly.
If it still looks the same like this then I would say there is a problem.