I don't think there is a broken corner case or paradox here, either in reality or in theory.
Consider what it means to actually have a very large resistor. Is it that weird to have large numbers of voltage noise in an environment where voltage does little? That is not the same as large numbers of current noise (since voltage noise is less than linear, RMS current noise must decrease in R) nor the same as large numbers of power noise (which should remain constant as R changes - I agree it'd be insane if power noise grew toward infinity with R).
Can you measure any pathological behavior of a big resistor with high voltage noise as the resistance approaches infinity? You can connect one leg of the resistor to ground and another leg to a measuring device, but what is the measuring device, a magical amplifier that somehow has even higher input impedance? From what I understand, voltage amplifiers with unreasonably high input impedance don't exist, even though they are generally desired, because they wouldn't function very well (for example, the thermal noise from the input impedance would be quite high).
If you leave the input of an actual op-amp buffer floating (i.e. at a near-infinite resistance to the nearest conductor), you could expect it to either measure an extremely random voltage, or to measure ground or rail voltage or something like that depending on the internal construction (because the input impedance i.e. impedance between input and ground/rail/whatever is not actually infinite, and in fact less than the impedance of some air in the room). The latter seems to be what happens in reality but I don't think the former would break intuition either.
Also, from a physical perspective, thermal noise doesn't care what the resistance is, it is just randomly adding energy to electrons. That should directly modify power noise independent of R. And as mentioned, power noise being constant in R is equivalent to voltage noise increasing in R.
I think it is easy to become really confused in electronics by thinking of voltage as if it were a fundamental quantity, just because so many circuits store and manipulate information in terms of voltage. Power is far more fundamental and generally does weird things wrt physical intuition less often ("wtf, a passive device can turn 10V into 1000V?!").