When an inductor is connected with a voltage source we get equal and opposite voltage on inductor against the source voltage.
That equal and opposite voltage gradually decreases with time which allows the current(caused by the source voltage) to rise gradually.
No. If the voltage source is an ideal voltage source, the voltage across the inductor remains constant. The voltage across the inductor causes the current to rise. If the inductor is ideal (zero resistance), then the current will increase linearly with time.
My question is what makes the equal and opposite voltage in an inductor to fall gradually
It doesn't in the ideal case, see above.
In practice, the voltage source might have a significant output impedance, which will cause its voltage to drop as the current increases. But note it's the increase in current that causes the voltage drop, not the other way around.
The inductor (unless it's superconducting) will also have some resistance. There will be a voltage drop across this resistance. Even with an ideal voltage source, the voltage across the inductive part of the inductor will fall, and so the rate of current rise will slow. Eventually, the current will rise to the point that all of the source voltage is being dropped across the resistance, and the current stops increasing.