In the datasheet of these encoders there is always a chapter mentioning masking-time and chattering.
I couldn't find any tutorial on this. What are these definitions about?
Chattering is the time during which the output is not stable when a transition from ON to OFF or from OFF to ON occurs. The signal quickly jumps between both states during a few ms.
Bounce (you didn't mention it but it is indicated in the datasheet) is the time, when the output is supposed to be stable in the ON state (between chatters), during which the slider doesn't touch the contact well, and the output is actually OFF for a few ms. It can happen when you move the slider, even if you stay on the contact area.
Masking time is the solution to that: it is the time during which you ignore transitions, to overcome both chattering and bounce. It can be done either in your software (e.g. waiting for the value to be the same over several samples taken at regular intervals), or using hardware (e.g. RC filter). This process is called debouncing, which is why your qustion is highly related to the "Debouncing buttons" question, in which you may find interesting solutions for implementing this.
Here is a diagram showing these effects:
t1 and t3 are the chattering times, and t2 is the bounce time.