The generic formula is:
$$V_{RMS}=\frac{1}{T}\sqrt{\int_{t=0}^{t=T} (V(t))^2 dt}$$
and for a sinus-shaped voltage:
$$V_{RMS}=\frac{1}{T}\sqrt{\int_{t=0}^{t=T} (V_0\sin(\omega t))^2 dt}=\frac{V_0}{T}\sqrt{\int_{t=0}^{t=T} (\sin(\omega t))^2 dt}=...=\frac{1}{\sqrt{2}}V_0$$
Now, it depends on how the PWM signal looks like. Let's say it just cuts away a part beginning at each zero crossing of the sine, then you get:
$$V_{RMS}=\frac{2}{T}\sqrt{\int_{t=p\cdot T/2 }^{t=T/2} (V_0\sin(\omega t))^2 dt}=...$$
(note: I have reduced the calculation to a halve wave, p is the fraction cut out by the PWM)
A more complex pattern would need to split the integral into all those intervals where the PWM is not zero. May be, your scope can calculate the RMS voltage numerically?
If the PWM frequency is very high compared to the sinus frequency (let's say 100 times higher), you just get
$$V_{RMSPWM}=p\cdot V_{Rms}$$
because the PWM acts in time intervals where the sinus is almost constant.
If the PWM period does not equal the sinus period, you can get something similar to a beat. But on average, the PWM gives the fraction of periods for which you have the sinus voltage or zero for a given angle. This also leads to a factor p.
