There is no real inherent distinction between breaking one side of a loop or the other side- it's all in series so breaking the negative or the positive side of the supply keeps electrons from flowing.
When you have an electronic switch and are turning off part of the circuitry with other circuitry it's easier to break the negative - called a low side switch (requires fewer parts), but if some other circuitry remains connected, that may cause problems. For example, if I break the ground connection on a module, the 'input' pins may start sourcing current back to the controller (because otherwise they would have to go negative with respect to 'ground' on the module)- it won't necessarily switch the current off completely and it may even damage the controller or module in some situations.
Look at a few of the many answers on this site where people have tried this, failed and a high-side switch has been suggested. It comes up quite regularly.
If it's something completely isolated like a relay coil, most designers will use a low side switch because it's simpler and there is no advantage the other way. In cars, the chassis is used as a return, so high side is preferred if the load is remote. Here is a useful document on automotive applications.
Speaking of automotive, there is one particular situation worthy of mention where removing the negative connection is recommended for safety reasons- and that is when you are working on a car. Since the negative terminal is almost always connected very solidly to the chassis, if you try to remove the positive terminal with a (conductive) wrench/spanner and the tool touches the chassis hundreds of amperes will flow, causing the wrench to get red hot. Some people have left their wedding or other rings on and receive severe burns (to the point of possibly losing their finger) when the ring formed part of the circuit.
So take the negative terminal off first and put it on last if you are working on a car. And remove jewellery.