For a prototype, with careful use, this would still have problems. The LM317 does not tolerate reverse voltages, but you can protect it by putting a diode across it, allowing any reverse current to flow through the diode to the other side (just the capacitor in this case).
In no case should the external power and the USB be connected at the same time.
The USB port should be capable of supplying 500mA at 5V without a problem, so it would appear to meet your needs.
In a production circuit, if you can handle a little droop on USB voltage, use a diode from the USB connection to the circuit's 5V rail. If the USB is connected and no external adaptor is connected, then the USB will supply up to 5V (after the diode's voltage drop). If the external adaptor is connected and not USB, then the external adaptor will supply all the required current at 5V. If both are connected, the external adaptor will be at 5V, while the USB will be lower, thus the diode will protect the computer from possible issues, and the USB won't supply any current.
It's not the best design, but it's quick, simple, cheap, and will save others from damaging their USB ports. There are many ways to handle this problem more elegantly, many using mosfets to switch the best available power into the circuit. But for a simple prototype, this isn't a bad start.