Fluctuating voltage on the analog input by itself is not a problem**. If you know that the sensor isn't connected, you can ignore the fluctuating value in the software.
** As long as absolute maximum values aren't exceeded, of course.
The O.P. doesn't say what type or model is the sensor (?). Nevertheless, it does say that the sensor output is voltage**. In this case, a pull-down resistor in front of the unity gain buffer would work without affecting the signal.
** I'll make the assumption that the sensor has voltage output and low output impedance. This rules out resistance sensors such as thermistors.
A pull-up resistor might be a more useful option. It may allow you to detect if the sensor is present or not. If the sensor is not present, the pull-up resistor would drive the analog line to a high voltage, which would correspond to a condition which you know can not occur. For example, suppose that the sensor is sensing water temperature. In the absence of the sensor, the pull-up resistor would generate a voltage corresponding to 300°C. You know that that can not occur, therefore you can infer that the sensor is not connected.
To test the effects of the pull-up or -down, measure the signal from your sensor first with the pull- resistor. Then remove the resistor. The signal shouldn't change.