I'm afraid that with that dinky little 100mA panel you will be sorely disappointed with the charging speed.
I've tried similar experiments with solar panels and you probably won't break your phone but it will probably refuse the charge altogether and give you something along the lines of a "charging was disabled to prevent damage to your device" error. Which is just a generic error your phone gives from over-voltage, under-voltage, or under-current.
For one, you need a 5V voltage regulator. Step down is usually more efficient than step up, so I'd go with series panels for 10V or 12V then step it down.
The second issue is you won't have enough power to charge a phone with just 100mA. The phone's charger isn't just a basic regulator. It's smart, and if you don't give it enough juice, it won't change slowly, it will straight up refuse to charge.
I'd suggest at least 5 to 10W of panels to just BARRRELY get any juice. Is it big? Yes. But it's also realistic from my testing tbh.
You will get more like a puny 1-2W out of quote on quote "5W" panel in direct sunlight in any realistic average conditions (so literally anything that's not an absolutely 100% perfectly clear sky, face meltingly hot summer day). Because even high efficiency monocrystaline solar panels kind of suck.
Also from my testing poly panels are even further off their rating and basically useless for anything other than a novelty/gimmick, or powering solar calculators. So I'd 100% recommend you go mono panels, despite the slight cost increase. And go for 5 to 10W total, you can do a combo of parallel and series in fact, to get the target voltage you want. Though keep in mind that one panel not getting as much sun can sort of "pull the others down"