What is the expected output of the solar panel, capacity of the battery, current capability of the 5V regulator, current required by the load?
It sounds like you need to lower the comparator from 4.0V to maybe 3.6V. The comparator doesn't tell the battery charger when to stop, it only tells the load when to switch on or off. All it needs to do is turn off the load when the battery gets too low, and not turn back on again until the battery gets somewhat charged up again, so hysteresis is good, but it shouldn't be trying to do battle with the charger over what happens at 4.0V. Put it at a different threshold so you can tell what mode you're in.
Mode 1: Panel providing enough energy for battery and load, battery is charging and between 3.6V & 4.2V (or is full and pegged at 4.2V), load is on. If the panel remains lit this way the battery never gets discharged.
Mode 2: Panel not providing enough energy for battery and load, but battery is making up the difference, battery is discharging and between 4.2V and 3.0V, load is on, and will turn off when battery discharges to 3.0V. This assumes the load isn't so huge that it immediately overwhelms the battery or 5V regulator when it turns on.
Mode 3: Battery has hit low-voltage threshold but not charged back up over hysteresis yet, panel is charging battery when lit, load is off and will remain off until battery charges up to 3.6V. If the battery doesn't charge at all due to the panel being dark, the battery charger may go into a low power mode and needs a certain input from the panel to turn back on again, see the data sheet.
Mode 4: Battery has gone above hysteresis voltage of 3.6V and load has turned back on. Battery voltage is determined by panel input vs load output. If panel > load, battery will charge until full (actually this is Mode 1), and if panel < load, battery will discharge until empty, or some variation between charging & discharging based on sunlight fluctuation. If you want the battery to get less wear, make the panel larger. If you want the load to remain off until the battery is fully charged rather than partly charged, then maybe feed the HBO pin into your comparator as a reset.
That's how it should work, but you have oscillation everywhere. You don't describe what the load is, and it sounds like it could be much larger than the battery or even panel + battery together can support. Either that or there's something wrong with the 5V regulator, which since it's the comparator rail could mean when the load is on, somehow 5V glitches, the comparator gets confused, the load turns off, 5V recovers, repeat. As far as I can tell the only thing the 5V regulator powers is the comparator, not the load? At the very least you need to replace the load with a simple resistor (say 1kohm) that draws small current relative to capabilities of the battery charger and battery capacity, until the rest of the circuit seems to work as desired. You might try pre-charging the battery on a bench supply (with both V & I limits set appropriately).
There could be other problems, like signal integrity, you forgot to add a detail from the data sheet somewhere, something is feeding back in an unexpected way. Please try the suggestions above and report more information about the parts not on the schematic and we'll see if we can help more. You'd probably get more people looking at it if your schematic was easier to read, like get rid of all the jumpers and optional parts and show how you actually have it wired, show the comparators as triangles with +/-, make the net names clearer e.g. VLOAD is really the battery voltage so call it VBAT+ or something (VLOAD would be the other side of Q2 where the load is), and so on. Maybe a photo of the setup.