Oddly the datasheet is doesn’t mention a maximum supply voltage (or current), but suggests a 3 or 3.3V supply so I suspect it’s out of spec on 5V. Adding a 3.3 supply would likely help, otherwise, depending on the current it draws, you might be able to power the device from an IO pin if you can spare one, and power it up only when you need to make a measurement. I don’t know if that would help but may be worth a try.
Eventually I desoldered all parts besides the controller from the PCB and got correct measurements from the humidity sensor.
So probably the sensor does not like to be so close to the controller. What puzzles me however is that putting the controller in power down sleep mode did not make any (noticeable) difference.
The issue can be reproduced in the test circuit on the breadboard. If the sensor is near the controller, the humidity reading slowly drops at least 5% to more than 10% below the actual value. Even the temperature measurement seems to be affected; it is 1-2 °C too high.
With the humidity sensor at least 3 cm away from the controller the readings are super accurate and stable.