First, you need an estimate of the heating power dissipated in the battery. That, in conjunction with thermal mass and thermal resistance to ambient will let you model the temperature of the battery.
Secondly, to estimate the heating power - I^2R - use an estimate of internal resistance and a measurement of the current.
The internal resistance can be estimated by comparing the open circuit voltage to the loaded voltage. The voltage difference between those two, divided by the load current, is the resistance sought.
Most loads can hold over when disconnected for a millisecond at a time, so to measure the open circuit voltage, the battery can be disconnected once a second for, say, a millisecond, and the open-circuit voltage measured at the end of the millisecond period, just before the load is reconnected. A mosfet switch can do the disconnecting job.
So, the steps are roughly:
- measure open circuit voltage
- measure under-load voltage and current
- estimate internal resistance
- estimate self-heating I^2R power
- update the thermal model with the self-heating power
- get the temperature from the thermal model - it’ll be one of the state variables.
These steps would have to be performed periodically, say once per second, and the thermal model updates will be fed the heat generated over a given sampling period - heating power * period. The under-load current and under-load voltage should be RMS values. When the loads are fairly steady, the instantaneous current sample will be close to the RMS value.
If the load is more dynamic, the internal resistance‘s dependency on loaded current and voltage will have to be sampled much more often than the open circuit voltage’s contribution to it. Eg. sample the voltage and current at 1kHz, and for one of those samples disconnect the load to get the open-circuit voltage (and a confirmation that the load got in fact disconnected!).
A fairly accurate single-cell battery thermal model would need just a couple of time constants. It can be extracted using any of the many available model extraction techniques.