For a simple astable oscillator \$\mathrm{T_{hi}=\ln2\cdot (R1+R2)\ C}\$ and \$\mathrm{T_{low}=\ln2\cdot (R2)\ C}\$, and \$\mathrm{T = T_{hi}+T_{low}}\$.
why the first one did not work?
If only R2 varies, Thi varies but also the period, T does. And the amount of change may not be enough to rate it as a duty-cycle change. Or at least, it may not be that distinguishable.
what is the role of the diodes?
is there a formula for this?
For the modified circuit (the one with diodes), the diodes determine which portion of R1 will be taken into account during charging and discharging of the capacitor, C:
During charging, D2 will be forward biased and D1 will be reverse biased. So only the left-side portion of R1 (let's call it R1a) will be used: \$\mathrm{T_{hi}=\ln2\cdot (R2+R1a)\ C}\$.
During discharging, D1 will be forward biased and D2 will be reverse biased. So only the right-side portion of R1 (let's call it R1b) will be used: \$\mathrm{T_{low}=\ln2\cdot (R1b)\ C}\$.
The period is:
$$\mathrm{T = T_{hi} + T_{low}=\ln2\cdot (R2+R1a)\ C + \ln2\cdot (R1b)\ C \\
\Rightarrow T= \ln2 \cdot (R2+R1)\ C}\
$$
And the duty cycle is:
$$\mathrm{D = \frac{T_{hi}}{T} =1-\frac{R1b}{R_{total}}\ \ \ \ \ \ ;\ R_{total}=R1+R2 }$$
As can be seen from the last two equations above, the duty cycle varies with the position of R1's wiper because T remains constant.
what is this mode called?
I don't know if it has a specific name. But still, it's an astable oscillator.