MII MAC Mode
This mode will be used when you want to connect MAC(Media Access Control) Devices which have inbuilt Ethernet Interface Control like Processor(say iMX28), Controller(say AT91SAM).These are capable to taking the data for processing and using it for different purposes.Other case is cascading i.e. connecting two or more KSZ8863RL together for increasing the number of Ports on your Switch.
MII PHY Mode
This mode will be used when you want to connect KSZ8863RL with external PHY(say KSZ8051RLN), in short the purpose of PHY is to convert the data coming from Processors or MCU to convert it in the format so that it can be transmitted over copper CAT5 cables and when the data is received from the CAT5, they convert it in the format that can be used by our Processors or MCUs.
Since you want to connect KSZ8051RNL which is a PHY, Physical Transceiver, you should use MII PHY Mode.