It is normal for these gas voltage regulator tubes to glow. The OC2 and OA2 are not vacuum tubes, but are filled with low-pressure neon (OC2) and argon (OA2). They operate in the glow discharge region, where a very small increase in voltage causes a large current increase.
The tubes should glow steadily, however, rather than flicker. As the regulator ages, the pressure drops as gas is adsorbed onto the electrodes, and even onto the glass shell through sputtering. As pressure drops, the glow discharge voltage increases. The flickering indicates either that the OA2 has aged, or that the old electrolytic capacitors are a bit leaky and the power supply cannot reach it's intended voltage.
As you suggest, the electrolytics age poorly and should probably be replaced. Using a slightly higher voltage rating than the OEM is a good idea, and 250 VDC for a 150 VDC unit is fine. Slightly exceeding the capacitance is also acceptable, but too large a capacitor could draw too much on startup, eventually damaging the rectifier.
Don't replace a multi-section capacitor with a single one, since each section is part of an R-C multi-stage filter and voltage divider. You can use a separate modern electrolytic for each section. The higher-voltage unit's negative connects to chassis ground, according to the schematic below, but the dual 150 VDC unit appears to be floating, so be careful placing that replacement so it doesn't contact ground. BTW, some rebuilders of vintage equipment put smaller new capacitors inside the aluminum shell of the old one for appearance, but there's no need to do so.
Once capacitors have been replaced, check the voltages at the test-points above the capacitors. If all is within tolerance, and the OA2 is no longer flickering, enjoy using the oscilloscope! However, if the tube still flickers and the voltage at pin 1 of the OA2 is a bit high, you'll need to replace the tube or use a 150 VDC Zener diode.
Use care with the high voltage, and use safety glasses -- should a vacuum tube shatter, particularly the CRT, glass goes flying at the speed of sound.