I2C is an open collector. What that means is that it uses a npn transistor to pull the line from open (off), to ground (on). A pull up is used to allow variation on the voltage that the bus works at. The resistance of your pull ups change at different speeds and bus size and wire distance because you are relying on the capacitance of the bus and the value of the resistor to change the state of the bus. It's an RC network and timing changes as the frequency changes. If your resistor is too weak, it cannot change the signal fast enough at the higher i2c speeds. If it's too strong, then the same happens and you cannot put all the devices on the bus you want.
The reason that pull downs are not used is because a ground referenced signal is easier than a VCC referenced signal, if you want to be able to use arbitrary voltages on the bus independent of VCC. A high side driver PNP transistor needs a base voltage close to the collector voltage, which makes it more difficult compared to the low side NPN driver.