Atmospheric conditions can influence the absolute accuracy but the 4 rovers are very close to each other (let's say in a range of 10 or 20 meters). Shouldn't all the GPSs on board be influenced in the same way?
Yes and no.
Yes, each signal from each individual satellite will induce the same satellite/receiver distance estimation error (due to propagation delay uncertainties and satellite position uncertainties) for all 4 rovers. If all rovers are using the same satellites to determine their positions, expect the position calculation errors to be similar, and the accuracy of their relative position to be pretty good. I'm assuming there are no significant reflections involved.
No, on stand-alone GPS receivers you have no control over which satellites are being used for the location calculation. You need at least 4 satellites, but 8 or more can be used if you have a good receiver and unobstructed sky view. In practice, the list of satellites being used in the calculation changes over time: satellites with borderline signal strength can be dropped or added to the list dynamically. If the lists of satellites used in two different rovers are different, you can no longer assume that the same errors will affect the position calculation on both receivers equally and the relative position accuracy will suffer as a consequence. Also keep in mind that the satellites are not stationary and over minutes new satellites will rise over the horizon while old satellites will disappear, what can make the performance of your system time-of-day dependent.
Technically, the best solution possible is to use RTK, as @AndersPetersson recommended. This technique was conceived to provide incredible relative position accuracy. Typically, you'll need a static reference station and a RTK-capable receiver on each rover.
A middle ground solution would be to use GPS assistance (A-GPS) if your receiver has that capability. The idea is to obtain correction parameters from the internet that will improve the absolute position accuracy of each individual receiver, what will indirectly improve the relative position accuracy among the rovers.