There should be no problem doing this provided that the battery is never charged above Vmax - typically 4.2V/cell.
Assuming that battery is a single LiIon cell then a safe method is to provide the PV input via a diode and to clamp the PV voltage (not the battery voltage) such that Vbattery can never be charged above say 4.0V.
Clamping the battery at 4.0V is unsafe if the external charge ever takes it above that voltage. Clamping the PV voltage with diode separation means that if the battery is above 4.0V it will not be PV charged initially.
To clamp the PV such that Vbattery can never exceed 4.2V - and so that it typically stops charging at 4.0V, then you either need to properly allow for diode drop at low current or provide a simple opamp based circuit that allows sharp detection of voltage.
One method would be to use a TL431 voltage reference with the ref voltage detect pin driven from the battery, but the TL431 used to clamp the PV panel. At low enough currents the TL431 can sink current directly,. With a higher current PV you may need to add a pass transistor.
TL431 datasheet
R1 R2 set PV clamp voltage
Clamp charging when Vbat reaches say 4.0V for single LiIon cell.
A disadvantage is that R2 + R1 provide a constant drain on the battery. They can be sized such that the current is insignificant or, if desired, a high side transistor on the divider can be turned on only when the PV panel is producing significant voltage.
simulate this circuit – Schematic created using CircuitLab