This isn't the usual rotary encoder pattern, but it should still work. See this wikipedia section for the normal gray-code implementation.
Basically, you will need to:
- Store the current state of the encoders to a register
- Wait for some length of time. (This must be short enough that only one transition occurs)
- Compare the new encoder state with the old encoder state to see if they are different.
- Based on your previous state, and the new state, decode the direction. (This step/algorithm would be easier if you used a typical implementation.)
The disadvantage of your setup is that it takes two transmitters, probably has a more complicated pattern drawn, and there's no way to identify if the encoder has gone 2 steps instead of 1. Additionally, only one bit changes with the gray code count, so the comparison routine is easier and not a look-up table.