As question says, I want to calculate the hysteresis of Reed sensor and I don't know how. As I seen in Reed sensor's documentation, hysteresis may be x/y, or the difference between x and y, and I'm not sure which is true.
You don't calculate the hysteresis of a reed switch sensor, you measure it, or read it from the data sheet.
What the diagram is telling you is that you have to get close with the magnet to switch on, but then it stays on until you pull the magnet further away.
The hysteresis has to be defined in a way that is useful to you. You could define it as the ratio of the distances, or as the absolute difference between the distances. The use in the data sheet of the % symbol indicates that they are defining it as the distance ratio.
Note that the magnetic field at the switch is not proportional to the distance of the magnet, so the ratio of distances is not equal to the ratio of fields required at the reed. If you were driving the reed switch with a magnetic coil round the switch, then field ratio would be more useful to you, and you'd define it and measure it that way.