Several factors make comparing the brightness of LEDs nontrivial.
The brightness of a light source depends on its spectrum.
The four different photoreceptors in the human eye have specific sensitivity curves over wavelength, and in aggregate the human eye is more sensitive to green wavelengths than to red or blue.
Wikipedia shows the spectral sensitivity for the human eye here:
Note that two different curves are shown: the green curve represents scotopic vision, which is when the eye is adapted for dark conditions, and the black curve represents photopic vision, when the eye is adapted to bright conditions, and more sensitive to color.
Because the candela is a photometric unit, it takes this spectral sensitivity into account--photometric units are based on the luminous flux of a light source, which is determined by multiplying the power spectrum of a source by a luminosity function representing the human eye's spectral sensitivity and then integrating, as opposed to radiometric units, which are based on the total radiant power of a light source, which is simply the integral of the power spectrum.
So comparing candela ratings should be apples to apples, but if you take another look at the graph, note that there is more than one black curve. These represent different curves established at different times. Because there are different curves, photometric measurements are only directly comparable if they were obtained using the same luminosity function, and unfortunately the specific function used is not often specified for indicator LEDs (it is more likely to be specified for high power LEDs intended for illumination). But for most purposes the numbers will be close enough for a rough idea.
Photometric measurements must account for angles
The candela in particular is a unit of luminous intensity, which means it indicates a certain amount of luminous flux (that is, radiant energy weighted by wavelength) within a certain solid angle. It's equivalent to lumens per steradian. This means that emitting a given amount of light over a small area will produce a higher intensity in candela than the same amount of light emitted over a larger area. It also means that the candela rating of an LED is only meaningful within a specified angle. So if you have two LEDs with the same wavelength and candela rating but different viewing angles, they should appear equally bright from directly-on, but as you increase your viewing angle, the the one with the narrower viewing angle will appear less bright than the wider one.
In addition, the intensity is not going to be constant through the entire viewing angle, It will tend to be most intense at the center, and will drop off to some degree towards the edges of the viewing angle.
Photometric measurements must account for areas
The brightness of a directly-viewed light source depends on the apparent size of that light source. A given intensity of light emitted from a point source will appear brighter than the same intensity emitted from a larger area, so a clear LED will tend to appear brighter than an otherwise equivalent diffused LED. The relevant photometric measurement here Luminance, which uses the units candela per square meter, sometimes called nits (which are in turn equivalent to lumens per steradian per square meter. Photometrics can be complicated!).
Note that the luminance will also be affected by how the LED is presented to the user. Lightpipes and lenses will change both the angle and the apparent area of emission.
Specifications must be given under known conditions
In order to obtain the specified performance of an LED, it must be used under the same conditions under which it was specified. Two LEDs of the same candela rating may have been measured at different operating currents, so if you operate them both at the same current, they will exhibit different performance. Temperature and age also play a role, but current is the main factor. Note that when driving an LED from a constant voltage (such as an MCU pin or something) via a resistor, the operating current will depend on the forward voltage of the LED which will depend on the LED chemistry. So two LEDs with similar specifications but different compositions will require different resistors to obtain the same operating current. This is fairly elementary, but it's easy to overlook when comparing LEDs of similar color.
