If the inputs are active it is more likely the isolators themselves, but in any case it is an unusual amount for the application.
Are you certain the temperature increase applies to the entire PCB ? It is not clear to me if you are reporting a local rise or overall rise.
Furthermore, an IR camera is not a very reliable method for quantifying thermal rise, although it is very effective for finding hot spots, it should not be relied upon to quantify the thermal change with any accuracy. It would not surprise me if the components on the PCB would show up brighter in the it camera after warming up.
5mA pullups at 5V is .025W of power, likely not the cause of the thermal increase of the whole pcb, you may observe some local increase of the component. The 1/8W rating is a maximum rating and you are currently at 20% of that rating.
Local heating of a passive component to a steady state is generally benign, sometimes it may change the value of the component in a circuit in a way that requires consideration but this is not a concern for a pull-up application.
Particularly warm components or small pcb can result in overall heating of the whole assembly. A good ground plane with many solid connections improves the thermal conductivity of the assembly. Mounting screws connected to this ground plane further extract the heat to the enclosure if necessary. A dedicated heatsink for the component can be considered if the plane cannot handle the thermal current.
Sometimes , thermal isolation or positive feedback can lead to a situation where the temperature of a component does not stop increasing (even possibly over course of many hours) leading to thermal runaway and catastrophic failure.
For more thermally sensitive and high power devices there is a universe of detail and detailed modeling available to consider so the above is just a brief overview that is appropriate for a basic digital design like the isolator design you presented.