Edit in progress. Ignore for now.
Use modulo arithmetic.
// Pseudo-code
// error
error = (currReading + 512 - prevReading) % 1024 - -512;
prevReading = currReading; // Keep track.
This code shifts the reading from {-512 - 511} to {0 - 1023}, subtracts the previous reading and gets the modulus. The answer will be in the range 0 - 1023 but will have a positive offset of 512 which is subtracted out.
Suppose my current position is 10 and setpoint is 1023. The error is 1023 - 10 = 1013 and the motor starts to move towards 1023 to decrease error.