Yes, regulator stability could be affected by the added capacitance.
Assuming your required current is relatively low, simply adding a series resistor to the much larger capacitor (or using an electrolytic type with high ESR) should ensure stability without causing too much voltage drop when the added cap is sourcing current.