From the page you linked for your picobuck:
This PicoBuck supports PWM control so long as the signal is above a minimum of ~1.5V and each LED is driven to ~350mA.
What might be happening here is that there is a capacitance (either an intentional cap, or parasitic capacitance...most likely a cap tho) between your square-wave PWM output and the corresponding input to your PicoBuck.
- When you turn down the duty cycle to get the really low outputs, it can take multiple pulses to sufficiently charge the cap to >=1.5V.
- This can effectively form another oscillator (similar to a sawtooth osc) where the cap charges for several pulse cycles before reaching the PicoBuck's threshold voltage & 'tripping' the PicoBuck to flash the LED, down to a low enough frequency to cause visible flashing as the duty cycle reaches low enough values.
Possible fixes:
- If there's an easily removable capacitor on your PWM output, remove it (or at least replace with one of significantly lower value).
- Try adding an NPN & a schmitt trigger/P-FET (or an op-amp with high voltage multiplier) between the PWM output & the PicoBuck's input to 'square up' the signal waveform to as close as possible to a clean +5V/0V square wave, low-impedance signal.
- Check for supply voltage irregularities in either your PWM controller's supply pins, or the same for the PicoBuck. I suspect that your notice of "it get[ting] worse when [you] use all 3 RGB components" is most likely that it's more noticeable due to proximity/visual comparison, rather that affecting the 'actual' output. However, simce you mentioned it, checking for supply voltage issues would be the first place to look for something that could cause that.