Without knowing exactly what "using a D flip flop" means, I can provide at least two approaches.
First, instead of a single switch, you can provide two - an up/down switch and a "load control" switch. Using an NO switch operating on the open-to-close action for the control switch can be done by
simulate this circuit – Schematic created using CircuitLab
But keep in mind that I have provided no debouncing on the clock switch, which you must provide.
Alternatively, if you want the control FF driven by the same clock which drives the counter, you can do it but you have to delay the clock between the control FF and the counter in order to allow a change in the control input to settle before the counter acts on it.
simulate this circuit
Note that a pair of inverters makes a perfectly usable delay, and multiple pairs of inverters make for longer delays.