You're going to be hard-pressed to compress all of this functionality into any smaller of a package. Let's go through what you're trying to accomplish:
Regulate down from wall wart power (most likely 9V-12V) to clean, regulated 5V
This is easy and could be accomplished a zillion different ways. Current draw and input voltage is what really influences your choice here. You can easily get a small package linear regulator, but if the input voltage is too high, you start needing bigger and bigger packages to handle the heat, and you can get up to D2PAK and still be throwing too much heat. Linear regulators handling high input voltages is usually sucky for any moderate output current.
In this case, you need to step up to a switcher so you can avoid these heat issues. As far as the smallest package/simplicity, I have used the TPS84250 from TI in a design. About 14mm x 14mm of board space and 7V-50V input with 2A of output current and adjustable output voltage. They are very expensive compared to the raw components (switching controller, inductor, diode, etc) at $10 - $13 per piece in low quantities, but we're talking about simplicity here, right?
There are similar switcher designs in the TI Webench design center (output current / board size wise) that can be built for much cheaper, but then you're using more components and spending more time on layout. It's going to be a trade-off.
Select between regulated 5V and USB VCC for input to our 3.3V regulator
There are also a few good ways to do this... mostly either using discretes (diodes) or MOSFETs. There are even some power controller ICs with the MOSFETs built in. Can't beat that for integrated/small. Again, a favorite part of mine is the LTC4415 from Linear Technology. This IC will OR two power sources for you with its integrated MOSFETs, and prioritizes one of the inputs automatically for you. It also lets your set independent output current limits for each output so you can configure the USB input to match your 500mA limit, and the wall wart current limit to match your switcher's output current limit, etc etc. Board space consumption here is pretty small.
Again, a little pricy... these badboys are like $3 - $5 in low quantities but they do make the prioritized power source requirement pretty straight forward.
Regulate down from 5V to 3.3V
This portion is pretty obvious. Find the smallest package size with enough output current. Optimize in pricing, etc. Done.
Other Considerations
I know you mentioned wanting to eliminate components, but don't forget that you still want system-level protection against component failures... i.e. a fuse in front of the switcher in case the switcher goes haywire somehow and short circuits. Same thing goes for USB power. Your device should be doing its best to play nicely with all systems and signals it integrates with.