Yes, a rotary encoder can be added to any motor, including a servo that has been modified for continuous rotation, and used for position control.
However, this is a bit counter-intuitive, since the servo's internal circuitry already provides position control, using the integrated potentiometer as a rotary sensor. Disconnecting this positioning mechanism, then adding an external position determination sensor or encoder to achieve the same result seems unnecessary.
Be that as it may, the usual method to obtain such position control with a rotary encoder involves bonding the rotary encoder to the shaft of the motor, then taking the encoded values into a microcontroller, using a PID algorithm in the microcontroller, driven as a function of this position information, to control voltage and polarity of the motor for bringing the motor to a halt at the desired position.
In very simplified terms, if the servo's detected position, courtesy the rotary encoder, is clockwise to the desired position, the PID algorithm would rotate the motor anti-clockwise until the two positions are identical. Vice versa for detected position being anti-clockwise to the desired position. If any external torque moves the shaft away from the rest position subsequently, the rotary encoder feeds fresh angular data to the PID code, which then applies the logic outlined above.
Regarding addition of a mechanical stop to a servo motor, this is a feasible approach in general - the motor will draw its stall current, which is higher than normal operating current. You do need to check the datasheet of the motor in question to verify how long it can withstand stall current, though. Not all motors can cope with constant stall.
The solution in such case can be one of the following:
- Use code in the microcontroller, along with appropriate current sensing circuitry, to detect when the motor current goes over a calibrated threshold, i.e. the motor stops at the mechanical stop. At such time, stop powering the motor. This is complex, and non-trivial at the circuitry end due to power losses in the sense resistor you use. For large current motors, Hall current sensors or non-invasive sensors can be used, reducing the power loss.
- Use a limit switch instead of a mechanical hard stop, and use the switch close signal as a digital input to your microcontroller, and use code to stop powering the motor when the limit is triggered. This is the simplest and least expensive solution, and safe for the motor as well.