How much current can a CAT6 cable reliably handle? I want to use 3 of the cores for +5V and 3 cores for GND. I'm wondering at what current I need to think of another power solution.
At 5V you'll probably run into voltage drop issues before you run into current limitations (if the length is more than a few meters).
Edit: If the length could be as long as 10m, and assuming AWG24 size-- resistance is nominally 84 ohms/km so 0.84\$\Omega\$/10m, so three in parallel, round trip, would be 0.56 ohm at 20°C. If 5% voltage drop (250mV) was acceptable, that would be a current of 440mA maximum, so maybe 350-400mA maximum allowing for temperature.
802.3at Type 2 POE limits the current to 600ma per "mode" (pair of pairs) which is equivilent to 300ma per core.
So if you assume the IEEE got it right then you can safely deliver about 900ma on a setup with three positive wires and three negative wires. I expect the IEEE were pretty conservative to allow for less than ideal installation methods and that with a single cable in free air you could go somewhat higher.
However that is not the whole story. As Sphero points out in a 5V system volt drop is likely to become an issue before cable rating does. He came up with a current of 440ma for a 10m length and a somewhat reasonable volt drop.
An obvious solution to this is to use a higher voltage supply and then step it down at the remote end. However this brings issues of it's own. It's not such an issue for "ghetto PoE" type systems because Ethernet is isolated but the fact that you are using three pairs for power (leaving one pair for data) makes me suspect that you are not planning to use Ethernet.
If you have a common ground for data and power interconnection and you raise the power supply voltage significantly above the signal voltage then you need to think very carefully about the impact of volt drop in the ground lead and also the impact of fault conditions where the ground is disconnected while the power and data lines remain connected.
It depends on whether cable is 24 or 23 awg, and what length of cable, and whether pure copper or copper around aluminium.
I would suggest between roughly 1.5 amps and 2.2 amps based on quality of cable based on googling limit for power transmission on 23 and 24 gauge wire, lowest end for cheapest copper clad aluminium 24 gauge wire.
Longer length of wire you suffer from voltage drop because only 5volt power, normally in POE applications they step it up to 48v and then step down again to 5 volt.
There are ways to go beyond that if you are willing to go to trouble. Take off outer layer and spread out the wires and they don't overheat as quickly and much more power can go through, counts as "chassis wire" rather than power transmission wire in look up tables.
You could also have 2 neutral wires, 2 wires at +5 volt, 2 wires at -5 volt, for some applications such as led lighting, and under balanced load would count similar to 10v with the current cancelling itself out in the neutral wires, similar to idea of house power being split phase 110volt/220volt in USA over 3 wires. That would give a 33% boost to power, and under nearly full/balanced load a big decrease to voltage drop, while allowing 4 different led lights to be individually switched on and off from the 6 wires, half the time the "neutral" would be the positive side on led.
Please, stick to the NFPA 79 Standard, Table 13.5.1, which specify the allowable ampacities for conductors with smaller sections (30 AWG to 10 AWG) than the NEC ampacities (14 AWG to 2000 kcmil).
This standard rates 2A for 24 AWG, 90 deg., single conductor.
And considering 8 carrying conductors inside a same cable, we should apply a 70% derating factor, with an estimate of 1.4A for 24 AWG, 90 deg., 8 carrying conductors.
This consider could vary according specific manufacturer certifications. Also, this assumes a CAT6 Cable as 24 AWG.