In your case:
(1) I'd establish hardware + software CURRENT drive proportional to PWM output value. Current and not voltage control is essential as lumens:mA is ROUGHLY constant but lumens:volts is very non linear. Very.
(2) I'd establish max current allowed per colour and
(3) brightness of each at that current.
Note that the measure is liable to be lumen/mA. As lumen are an eye response based unit, even if LEDs had equal light_mW/mA the lumen values per mA would differ. Also - when mixing coulours, even when the sun of (Red)lumen + green_lumen + blue_lumen) is constant as magnitudes change, you may well find that the perceived brightness varies. However, when two LEDs are observed one at a time, a difference in luminous output of nearly 2:1 is needed before a brightness change can be discerned by most people. Side by side maybe 1.5:1. Wall washing adjacent - maybe 1.2:1.
(4) Then I'd establish brightness per PWM step.
This allows desired brightness per PWM step to be used to select an RGB mix.
You could
scale in software so all colours use value say 0-100 for 0-100% brightness, or
0-100 for absolute intensities (setting most sensitive colour to 100 at mx brightness so it did not get overdriven OR
use data "on the fly".
eg SAY
Red = max brightness 200 units at 25.0 mA
Green = max brightness 120 units at 20.0 mA
Blue = max brightness 80 units at 20.0 mA
So Red = 8 brightnesses per mA, 25 mA max
So Green = 6 brightnesses per mA, 20 mA max
So Blue = 4 brightnesses per mA, 20 mA max
There are various ways of controlling this.
If you ALWAYS want top be able to use equal RGB then blue (being less max bright) red sets the limit at 80bu, 20 mA.
So Green max = 80bu/6 ma/bu= 13.333 mA max
Red max = 80bu/8 = 10 mA
Blue = 80bu/4 = 20 mA (as above).
You can use these scaling values on the fly OR if say PWM max = 250 = 25 mA then you could use brightnesses 0-100 per colour so that:
Red 100 = 10 mA -> Scaling factor = 1
Green 100 = 13.333 mA -> scale values UP by 1.33
Blue 100 = 20 mA -> scale values up by 2.0
So a value of say RGB 50 50 50 produces "white" light with equal RGB brightness.
Actual PWM values are 50 67 100.
Actual mA are 5 6.7 10.
So also 25 25 25 brightnesses yields white at half the above overall brightnesses and currents.
Changing to 50 20 80 or to 50 80 20 produces a new colour and APPROXIMATELY unchanged brightness IF independent lumen values are added. What the eye-brain system sees may differ.
How did your original experimenting work out?
I'm about to try 'playing' with some LED strips that I acquired.