Managing Li-Ion-based (and the particular LiFePO4) batteries requires some more sophistication than just "boosting to 14.4V". The charging requires CC stage, CV stage, and proper cut-off. While it is not exactly a rocket science, some electronics is required to do it properly, and portable solar-powered uninterruptible supplies have a good focus in power industry. Googling for [solar-powered UPS] gives about 25 million hits, and device cost is usually at $50 and way up, this is not a $5 project.
But theoretically you have a 60W OEM charger. I assume that your battery (which you prefer to hide from public) uses 0.5C charging rate, and therefore likely has 8AH capacity and gets charged in 2-3 hours. Your solar source has 20W in a good sunny day, or about 4A at 5V, so even at usual up-conversion practices it could deliver 1 - 1.2A of current at 14.4V. Which is not that bad, and can't be considered as "low-current". So your concern about hurting the battery is not well founded, but it will take 3 times longer, and the average daytime might be not enough to fully charge your battery.
Practically, if you have a limited solar source, the battery should be sized properly to utilize the daytime energy. More, using a solar source with fixed 5V converter is very suboptimal for 14.4 V output, so you better consider re-designing the solar panel to make a dedicated 4S batter charger. Or go to AliBaba/eBay/Amazon and get the whole professionally-designed controller for $39.
Or go to upscale professionals for $550. Other options are available.