The accuracy of the LM4040 is guaranteed with a 100uA current and typically changes only 300uV for a current change from 75uA to 1mA. The guarantee is +/2mV at room temperature and +/-15mV over a wider span.
Self-heating will affect it too.
So if you use even a modest precision op-amp you won't affect the initial accuracy or drift much. The antediluvian OP-07 has guaranteed Vos of as low as 75uV (which is << 2~15mV) and a guaranteed drift of less than 1.3uV/°C compared to the +/-31uV typical drift of the best LM4040 at the optimum current.
There are much better op-amps available than the 1975-era OP-07 in 2020.
So, yes, buffering it if you expect large changes in load will help, and it will reduce self-heating drift and changes with source voltage even if the load is fairly fixed. You can also low-pass filter out some of the LM4040 noise, which is probably around 200-250uVp-p.
That will also allow you to operate it at an optimum current, with a large series resistor (perhaps from a relatively high voltage supply as in your example) so the current is closer to constant.
You should really add up all the sources of error and compare it to your target, to see what is necessary or desirable.