Firstly, a disclaimer: I am not a medical professional, nor do I have any professional expertise in the area of ophthalmology. I'll try to leverage my understanding of failure mechanisms in sensitive sensor systems and some outside sources to venture an educated guess:
According to this summary from an ophthalmology journal, mechanisms of damage to the eye can be categorized as photothermal, photomechanical, and photochemical. For each mechanism, we should ask what the relevant time constants are in order to understand whether the risk for eye damage would be correlated with the peak (on) brightness or the brightness as you see it, averaged over e.g. a PWM cycle.
Photothermal - this occurs when the temperature of the retina is raised by incident electromagnetic energy. The thermal time constant of the retena is likely to be on the order of seconds (my guess, based on scale and thermal conductivity of biological tissue), so that average and not peak radiance would correlate to damage. At any rate, photothermal damage is observed in exposure to very high irradiance level (e.g. lasers) and not a likely risk with even the brightest incoherent LED.
Photomechanical - this occurs when compressive or tensile forces generated by incident energy cause mechanical damage to sensitive optical structures. If these type of stresses can arise on a very small mechanical scale, there might be some concern that the relevant time constant could be below the PWM period of your LED. However, you can probably rest easy, since the article associates this damage mechanism with irradiance in the range of terrawatts per cm^2.
Photochemical - this is the most common type of retinal damage, associated with e.g. looking at the sun. The chemical mechanism is ultimately oxidative - electrons in chromophores get excited by incoming light energy and can occasionally generate free radicals which go on to damage a variety of sensitive tissues. In another summary article here, a discussion of retinopathy caused by viewing a microscope or opthalmoscope with irradiance of ~1W/cm^2 provides some relevant numbers and references. At this level, damage is indicated on time scales in minutes to hours. To me, this suggests that the relevant biochemical processes are much slower than a PWM cycle.
As a final thought exercise, consider that many humans routinely glance at the sun for probably hundreds of miliseconds without suffering solar retinopathy. It is only when people resist the biological impulse to look away and hold their gaze for seconds or more (because they are checking out an eclipse, for example) that damage occurs.