Yes you can use that solar panel. It will have excess power capabilities in sunshine, but that power goes nowhere.
Note: You may still need to put two panels in parallel for cloudy seasons.
The max possible current from the panel is not destroying the charger under normal conditions. An analogy: A car battery can give 100 Amps, but still it doesn't blow a 12V 1.2W glove compartment lightbulb.
Instead, with the excess possible power, the panel can maybe drive two chargers in parallel.
Yes that panel might destroy your charger under bad circumstances such as bright sunlight in combination with battery full so the input voltage reaches "open circuit" due to no panel load. Maybe not by "overcurrent", but components inside may not be rated for higher voltage and burst or at least deteriorate.
You solve this issue by "clamping down" the voltage by means of Zener diodes.
I looked at my local distributor's webpage and they have 7.5V 5W diodes for c:a 0.4Euro each. 7.5V is a reasonable voltage for you, I think. If you are cautious maybe you can select 6,8V. Those apparently cost 0.5Euro each.
3 of those 5W diodes can together take 15W, thereby protecting your circuit without burning up themselves. You put these 3 diodes in parallel with your charger's input. Remember to leave long wires when you solder them, it's their cooling medium. Zener diodes are to be mounted "backwards", you can see it on the internet.
These 7.5V diodes work so that at and above c:a 7.5V they will "suddenly" decrease more and more in resistance, effectively stopping a raise in voltage, since more volts will load the source (your panel) with more amps.
I have been using 12V panels and have found that their peak power output is at around 16V. This suggests that your 6V panel works best at 8V (dangerous for your build!). You can achieve keeping those 8V by purchasing an "MPPT" circuit, but allegedly the gain is only 10%. The MPPTs I use are combined with a "buck converter" that can be set at a convenient 5V output, giving you a steady voltage both for panel and charger (if the sun shines).
However, without the MPPT, this best performance at c:a 8V suggests that you should have the diode cutoff as high as possible, i.e. at 7.5V. Which, well, may be dangerous.
So I suggest you buy 3 of each (6.8V, 7.5V) and try them out.
Also remember that allegedly best charging is only during 4 hours around 12:00. Slanted light and lower sun decreases power rapidly. So those 4 hours define in practice your charging time, or "charging budget". I.e. will those (milli)amps during those hours be sufficient?
(Note: this is written for laymen like me.)