So basically, you're just going to do power over UTP, not power over ethernet. PoE is a specific set of standards designed to safely supply a certain maximum amount of power from a PSE to a PD, whereas you are going to omit the protocol entirely and just running power. Correct?
In that case you are only limited by three things:
- Voltage rating of the jacks and cable (CAT6 cable is usually already extremely good dielectric stuff, often PTFE, so you can run hundreds of volts over that)
- Current rating of the cable and isolation transformers
- Noise of your PSE
You won't need to worry about ratings of the signalling electronics because they are fairly aggressively clamped to counteract insertion transients and of course they are AC coupled through a transformer. The only thing to worry about on the receiving end is that you have a dc/dc converter that converts the power down to a voltage suitable for your application.
You are injecting common mode current into the wire, and this has to go through the common terminal of the isolation transformers, through the transformers themselves and into the UTP cable. This means that your average magjack won't work - you need pretty beefy isolation transformers. This is easy to find in 100Mbit form, bit harder if you want gigabit communications (try shopping at Würth for those, but avoid them for as much as possible as they have the worst 'active marketing' department of any electronics company). I've designed a very big PSE in the past using these transformers.
Another, probably cheaper, way of doing things is just to use standard 100Mbps comms over two wire pairs and throw the power over the unused pairs. This saves the ~7 dollar expense on harder-to-get magnetics. You won't be able to reduce the wire diameter by removing those two pairs anyway, because you're bound to UTP anyway. the outside shell will stay the same no matter what you remove from the inside. You still can't use magjacks for obvious reasons.
Last thing to make sure is that you are not throwing any excessive noise around in the frequency domain of Ethernet. Ethernet has significant signal bands between 2 and 125MHz, so you need to make sure that you very much attenuate anything above say a MHz. You should be able to verify this with even the simplest oscilloscopes.