Well, ~1.6 Watts with a thermal resistance (junction to ambient) of say 6040 Degrees Celsius per Watt makes it go from 25 degrees ambient to ~125~90 degrees which won't really kill it, but that assumes your calculations are perfect. The heat you see here is actually ALWAYS being dissipated while operating in your 4A load condition.
The absolute max ratings is 150 degrees, so during the 4 microsecond switching period, if it manages to generate a lot of heat, enough to bring the junction up to 150+, then it may begin to fail. Seems a bit of a fair stretch, especially if you only switch it a few times. If you are switching many times a second, this may build up average heat enough for it to fail fairly quickly.