Statistical physics which describes what heat actually is gives to us also the equation P=kTB for how much noise power a resistor generates over bandwidth B in absolute temperature T. Factor k is Bolzman's constant.
We can make the Thevenin equivalent of a noisy resistor which has resistance =R. It's an ideal "noiseless" resistor R in series with a voltage source which has random RMS voltage sqrt(4kTBR). That contains your problematic 4. The next story explains the 4.
"Generates power kTB"? That noise power should be really available to a load. Only a matched load =R takes it all. If a voltage source with internal resistance R outputs power P to load R the voltage drops 50% due the voltage division in resistors. Thus
P = ((Un/2)^2)/R
If you insert P=kTB and calculate Un you get Un = sqrt(4kTBR).
The 4 is nothing physical, it's caused by our modelling idea - we made Thevenin equivalent. Its source voltage must have that 4. Otherwise it would not generate the wanted power kTB to a load which can take it all.
If a resistor is not connected to a load the full noise voltage sqrt(4kTBR) in theory exists between its terminals. Measuring it is totally another story because noiseless measuring equipment do not exist. I skip it.
ADD1: By watching removed answers I see that the main idea of my answer is already presented several hours earlier in a comment by user V.V.T
ADD2: In an ideal transmission line which has impedance Zo and the line is infinitely long, but there's a thermal noise source in the beginning, the propagating noise power over bandwidth B is kTB. That, of course assumes that the noise source radiates waves which perfectly fit into the line (=matched).
There's noise voltage over bandwidth B =sqrt(kTBZo). There's no 4 because this is not the Thevenin equivalent of a noisy resistor, but a transmission line.
If the line had finite length and it was properly terminated with an ideal noiseless resistor (=zero kelvin) =Zo that wouldn't change the voltage (over B), it would still be =sqrt(kTBZo).
Normal resistors =Zo at both ends would send noise kTB over bandwidth B to both directions. The noises do not correlate. The total RMS voltage over bandwidth B would be sqrt(2kTBZo).
BTW. If you can understand university level math and stochastic processes you can find academic writings of thermal noise. Here's one relatively new way to get the basic formula + some discussion of finer details such as how quantum mechanics is taken into the account https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.693.1026&rep=rep1&type=pdf