I prefer shift registers to multiplexers, but would think it would be practical to feed 64 encoders into sixteen 8-bit shift register chips (e.g. 74HC597) and have a small ARM poll those and repeatedly output the resulting counts (perhaps via MIDI system-exclusive events or other means). If the shift rate is 2mbit/second, that would yield a polling cycle of about 15,000Hz which should be adequate. At a 16Mhz CPU clock rate, that would allow about 64 cycles per byte, which would keep the CPU quite busy but be should be workable with clever code.
A key trick to getting good performance would be to use "sideways computations". If variable A0 holds the first input from each of 32 encoders and B0 holds the second input from each encoder (in the same sequence), and one performs the computations:
delta = B0 xor A1 xor B1
A1 = A1 xor (delta and A0) // Same sampled value must be used here and below.
B1 = B1 xor (delta and not A0)
then B1:A1:A0 will hold 32 three-bit grayscale counts. If one then performs
the computations:
delta = B1 xor A2 xor B2
A2 = A2 xor (delta and A1)
B2 = B2 xor (delta and not A1)
The approach may be extended to arbitrary depth, but a key observation is
that one need not process later stages as often as the earlier ones. One
could thus maintain arbitrarily long counters without having to increase
the amount of computation work per scan cycle.