Ok, let's consider some ballpark numbers. To begin, we need to determine what is the real capability of the UPS. A quick Google search reveals that this model uses TWO 12-V batteries, each about 7 A*h. So to start, you can't use only one battery, you would need two of them.
Now, 24V times 7 Ah is 168 W*h total capacity. Manufacturer's website says that 15-min run time happens if the load is 300 W, which means that the UPS is only about 50% efficient, which is not that impressive.
If you want to increase the run time of this device from 1/4 to 6 hours, you need to increase capacity of your batteries 24 times. This means you need two 12-V 170 Ah batteries, so you got the necessary capacity of individual battery about right.
Now, the UPS manual says that it takes 6 hours to charge the original battery. This means it will take 6 days to charge your new battery. I see no problems with this, since the lead-acid batteries are charged with constant-voltage, and usually LA batteries do not suffer from "overcharging"(unlike Li-Ion).
Regarding the heat, if this UPS can sustain 15 minutes of 300-W load, it likely can sustain 6 hours as well. However, since the efficiency of this UPS is quite low (their claim is 72% at full load), the dissipated heat is quite substantial, so you might need to add some blower to your re-designed UPS.