Your question assumes there is such a thing as absolute potential. It requires that, somewhere in the universe there is a place "X" where electric charges have no potential energy at all. In other words, there exists no other place in the entire universe that charges could move to, where they they have less potential energy than they do at "X".
The only reason an electric charge would move (that movement being called "current"), is because there's a point where it would have lower potential energy, and to get there they don't have to climb a potential "hill" first.
Think of it like a ball on a hillside. As long as the ball is sitting on a slope towards a lower gravitational potential energy (otherwise known as "downwards") it will be accelerated in the direction where the downward slope is greatest, and will only settle when it arrives at a point where the slope in all directions is upwards.
There may be a valley next door whose lowest point is below the ball's current position, but if it has to climb another hill to get into that valley and settle there instead, the only way it can do that is if it gets pushed up and over the hill separating them, by some external source of energy.
There's always a valley somewhere else, whose lowest point is lower than where it is right now. How low can the ball get? Sea level? But some land is below sea level. What about the bottom of the Kola Bore Hole? What about the centre of the earth? Supposing the ball could reach the centre of the Earth without damage, what about the center of the Sun, would the ball have less potential energy there? You see why it's not reasonable to say that potential energy, and by extension voltage, can have some absolute value. It's always relative to some other place.
You told us to neglect the voltmeter. In doing so, we assume that point 1 is physically disconnected, electrically isolated from Earth. In the absence of any means by which a charge at point 1 or 2 or anywhere in the circuit can travel to Earth, what you have created is a huge hill between those two locations. You've also relinquished any say you have in the question of which of the two valleys either side of the hill is "deeper", because there are other influences such as capacitive coupling to nearby mains wiring, which have suddenly become far more significant factors shaping the "terrain".
Sure, the Earth may have a lower potential, or it may have a greater potential than point 1, but you'll never know. In making the measurement, by inserting a voltmeter, you provide a route for charges to move between those locations, effectively removing the hill between them. Measurement changes the voltage "terrain". By joining the two locations electrically, do we raise the Earth's potential, or do we lower the circuit's potential? It's all relative, and the question is moot. In the end, all you need to know is now the Earth and the point 1 have been joined.
In your circuit, neglecting the voltmeter, the only thing you can say that has any meaning, is that point 1 is five volts higher in potential than point 2. Any other statement referring to the Earth or anywhere else ouside the circuit is meaningless (finite electrical resistance of the air notwithstanding) .
When you put the voltmeter in place, you join the Earth to point 1, providing a path that charges can use to move between them. However, will any charges make that journey? Is there an potential difference to motivate those charges to flow? In the absence of any other forces at work (like electromagnetic radiation, noise, and so on), the simple answer is no. There's no reason for any charge to move from point 1 to Earth, or the other way.
If you want charges to move between Earth and point 1 you have to make them move! You have to explicitly impose a potential difference, so that charges at one place find themselves on a potential "slope" that they can "roll" down. You have to do what you did to move charges through your 5Ω resistor, use a voltage source to impose a potential difference across it. If there's no such voltage source, between Earth and point 1, just the voltmeter, then there's no potential difference across the voltmeter, and the potential at Point 1 is the same as the potential of Earth.
Edit: In the parlance of somebody who teaches electricity to 10 year-olds, where generally you must show them that there needs to be a closed loop for current to flow "around", that last paragraph amounts to saying that the addition of a voltage source across the voltmeter creates just such a loop. Current is now permitted to flow through the voltmeter, and the presence of a voltage source in that loop motivates the charges to move. It's quite naive, though, because it lacks many elements of the full picture.
Edit2: Where the pressure analogy is concerned, there is such a thing as zero pressure, it's the vacuum, and therefore you may quote pressures relative to the vacuum if you wish. Typically though, that's not very useful, given that locally flow is dependent on the difference in pressure between two points, and has nothing to do with the vacuum, per se. I have always found the analogy between water and electricity to be wanting in a several respects, this being one of them.