The first thing to do is to figure what the fastest toggling speed of any of the lines will be. Find the fastest shaft speed you need to handle, then figure out the toggle frequency out of the encoder for that.
Once you know the fastest response time you need, this problem no longer has anything to do with a encoder. It's really about how stiff a pullup or pulldown needs to be to float the line to the released state when it is no longer actively driven to the opposite state. This is mostly a RC time constant calculation.
Let's say you decide that there will be no more than 100 pF of parasitic capacitance on a line, and that the fastest toggle rate is 10 kHz. Each cycle is 100 µs long, so each level is 50 µs long. If you need to decode two of these lines in quadrature, one should be well settled before the other starts to change. Let's say you therefore decide you want each line to settle to 90% within 10 µs.
90% settling happens in 2.3 time constants. One time constant is therefore (10 µs)/2.3 = 4.35 µs. The minimum pullup or pulldown resistance is therefore (4.35 µs)/(100 pF) = 43.5 kΩ. That's actually rather high. Unless this is a particularly low power application where you need to conserve 10s of µA, I'd just use 10 kΩ in this case.
Note that most of these devices have open drain or open collector outputs with a common ground. You would therefore need pullup, not pulldown, resistors. Check the datasheet to make sure you are using the correct polarity. High speed devices usually drive both ways and don't need pullups/pulldowns at all. Again, check the datasheet.