TLDR; Yes, but it is always a good idea to put a smoothing capacitor across the Vcc/Ground pins of the IC, and try to put that capacitor as close to the pins as possible.
The USB standard specifies that a "charging port" should be able to supply up to 500mA at 5 volts...
The voltage required by the chip depends on what speed you are running at, but even at the maximum speed of 20MHz it only needs 4.5 volts...
...so you are good on voltage.
The chip is also spec'ed to pull an absolute max of 200mA of current...
...(and that would mostly be from driving output pins), so you are good on current.
So, at least in terms of voltage and current everything looks good, but there is also the issue of noise. This noise can come from either the power supply or can be generated by the chip itself as it makes very fast changes to the amount of current that it requires (for example, at the instant you switch a bunch of output pins from 0 to 1).
As far as the noise coming from the charger, all USB chargers are not built equally. I tested a few crappy USB chargers to see how noisy the output was...
Here is one of the worst...
So the real concern here is that some of that noise (the jigglyness of the voltage) could cause the cause the chip to glitch.
A filtering capacitor across the power leads can help smooth this out. It is typically recommended to put a couple of uF of capacitance across the power leads where they enter your circuit to filter this kind of noise.
It is also generally recommended to put a smaller decoupling cap across the power supply pins, and to put this cap as close to the chip as possible to smooth out the very quickly changing power supply demands of the chip.
Much has been written about how to pick the correct decoupling capacitor size and type, including...
Atmel AVR042: AVR Hardware Design Considerations: Providing robust supply voltage, digital and analog.
Choosing and Using Bypass Capacitors: Application Note
...but a good guess for decoupling cap for a chip like this might be about 0.1uF.
All that said, in practice I've found AVRs to be remarkably tolerant when it comes to power supply, and I've also found name brand USB chargers to be high quality and smooth. You can often just use what you've got and putting, say, a 1uF across the power supply pins will usually eliminate any problems- especially if you are just bread boarding and not designing a commercial product. It is also a good idea to use the on-chip watchdog to reset the chip just in case you ever do have problems due to power supply issues (it also saves you from other problems too).