The potential of any point is a measure of the potential energy of a packet of charge at that point, relative to an arbitrary reference.
When a packet of charge moves from one point to another, it absorbs energy if going to a higher potential (charging a battery or a capacitor say), it releases energy if going to a lower potential (for instance heating a resistor).
Only differences of potential, that is voltage differences, are observable.
The conventions choose the size and polarity of the charge packet used to measure the energy. By convention, in electrical engineering, we use one coulomb of positive charge to convert between volts and Joules. We use positive charge regardless of the physical type of underlying charge carrier, whether it be holes in a semiconductor, electrons in a wire, ions in a plasma, protons in ice. Particle physicists use the charge on one proton to convert between volts and eV.
If we used a packet of negative charge to define the potentials, then our labels 'higher' and 'lower' would be reversed, however the world would still work just fine, as would all the mathematics as long as we applied the new convention consistently.