I did exactly as that picture showed, and I guess how it works is that it charges the AA batteries and then takes energy from them to charge something through the USB hole? Right? (I'm new to electronics), but when I plug in something (my phone charger) it doesn't give a message saying it's charging.. However, when I connect a USB powered fan with LEDs, it starts spinning..
What is the problem?
You did a lot of handwaving. A picture is good, but relying on it for all pertinent information is not good. I'm going to assume that is a USB A connector (like a host), and that the black rectangle on the right is a solar battery with the cells on the other side.
This is a very simple circuit. The solar battery is sized to charge the rechargable batteries when in sunlight. The batteries then supply roughly 5 V to the USB jack.
It makes sense this works with a dumb device like a fan that also only connects to the USB power leads. Apparently your phone is smarter than that and tries to follow the rules of USB. It may be trying to communicate with the host before it draws more than 100 mA, as required by the USB standard.
It may also be expecting its own proprietary charger that probably puts specific voltages and/or resistances on the data lines. When the phone sees this particular voltage/resistance signature, it knows it's hooked up to its proprietary charger and that it can draw some large amount of current. Most likely the real charger is capable of more than the 500 mA maximum a normal USB socket can provide. The special signature of the real charger is for two reasons: To let the phone know it's OK to draw more than 500 mA, and to make it difficult for third parties to make chargers. The latter is necessary since a charger could be made quite cheaply and the company that makes the phone wants to make lots of money selling simple and cheap electronics at obscene markup.
The problem is that you are supplying the USB plug with the batteries, which give about 4.5V total (the panel is biased at the same voltage). Since the fan is a simple load, it'll just spin at the speed it can go (be careful with the current!).
The cellphone is more picky about the charging voltage, and depending on the model this can be more or less strict. You should check the specs for the phone charger, and eventually build a step-up to match the requirements.
The voltage rating of the panel is indicative, and it will be satisfied only on determinate lighting (above a minimum level) and load (open circuit or very low current) conditions. Who determines the voltage is the battery pack, and in this case it's 4.5V. Then, the panel will give a current which is defined by a curve like the one below.
You can for sure use a 4 batteries pack, but then your panel won't be biased near the Max Power Point (MPP), which usually is around 70-80% of the maximum voltage (in this case around 4.5 V).
So you can use one more battery and then lower the voltage to 5V, but you'll receive less power from the panel; or you can use 3 batteries and step up.
What kind of device are you trying to charge?
Several annoying companies, like Apple and Sony want you to only buy their own brand chargers to charge their devices. This is mainly so they can make a little bit more money, but they spin it as they need to ensure that the charger is compatible.
So what they do is to make their charger a little bit non-standard, and the phone or whatever detects this, and will refuse to charge if it doesn't.
So, what's the difference? Depending on the device you want to charge, you will need to add some resistors between the power lines and the D+ and D- lines.
See this article about Minty Boost charger for more details.
Alternatively, it could just be that you've only got 4.5v, rather than 5.0v.
