You are only considering one small use case which happens to work for you. USB is a primarily a bus and devices must be designed to comply with specs to play nicely with other devices and hosts.
A USB port is only guaranteed to have at least 100 uF of decoupling so your 5,000 uF cap will momentarily take down the hub's supply and the hub and all the devices connected to it will reset. For this reason, USB devices are required to have no more than 10 uF of capacitance on VBUS.
It is possible to use a 5,000 uF cap for your device, but only 10 uF can be directly connected to VBUS. You will have to additional circuitry to limit the current to charge up the large cap to no more than 500500 mA 100 mA, until after you've been granted higher current by the host.
You could also design the device to differentiate between a USB power supply and a USB host port. If it detects a power supply, then it can connect the 5,000 uF capacitor.
A brief outline of electrical specs are here: http://www.beyondlogic.org/usbnutshell/usb2.shtml