That is, for example, filling the top layer of a 2-sided board with ground and the bottom with power rail.
Will this form a big decoupling cap across the board, or will it give me horrible crosstalk?
It does act like a big capacitor but its not a large value unless the PCB is fairly thin. The main reason you fill is to create the lowest resistance and inductance distribution for your power and ground. That way they are more stable when the load changes.
I had a board where I forgot to fill so the power and ground were distributed in 10mil trace. I had some comparators powered off the rails. The moment the comparator switched state, the power rail bounced causing the other parts to change state. next rev I filled all available area with power on one side and ground on the other and it didn't move more than a mv or two.