The problem with a single 10k pot in series with the group of 50k pots is that the point where they join is not 'stiff', is not a low impedance like ground. This means that adjusting any of the 50k pots will, as you've found, move that point and so alter the other oscillators.

The oscillator configuration you have chosen is amenable (to some extent) to voltage control, as well as resistance control, which allows you to have a low impedance solution.

<!-- Begin schematic: In order to preserve an editable schematic, please
     don't edit this section directly.
     Click the "edit" link below the image in the preview instead. -->

![schematic](https://i.sstatic.net/O7lmK.png)

<!-- End schematic -->

R1 will at best be a limited trim on the oscillators' frequencies, as you will have to keep Vref well below the charge voltage from your oscillator UA1. The control law will also be 'whatever you get'. However, the opamp buffer output is a low impedance point, and so will isolate the adjustment of R2 from R3 and any of the other oscillator controls.

Ideally, you would use voltage controlled oscillators. The type you have chosen is very prone to frequency variation with voltage and temperature variations. If you run the synth from a power supply, that may well have stable enough rails that Vref can simply be a resistive divider down to the pot. If from a battery, then you will probably want to use a voltage reference for stability.