Will WS2812B overheat because of epoxy?
It may or may not overheat. That depends on the amount of power which it will be dissipating. That in turn depends on the amount of light which you will be generating.
[The LED strip with WS2812B] will be glued onto wood and then covered with epoxy from the top so that all the components of the strip are in the epoxy layer which is about 2cm thick. The side are made out of wood, so there isn't any thermal runaway either. This method would make it also unreplaceable.
The wood onto which the LED strip is glued on is a bigger impediment to heat sinking, compared to the epoxy on top. A surface mount IC sinks most of the dissipated power through the legs into the board (less so through the case to the ambient air). Where will the board dissipate power? Wood is a relatively poor heat sink: low thermal conductivity, low specific heat.
If you can stick your LED strip onto metal and expose the other side of the metal to ambient air, then you'll be able to dissipate more power.