Yes you can use normal CAT5 ethernet cables for other things. These are cables with 4 separate twisted pair in them, with a reasonably controlled impedance and a different twist pitch for each pair so that crosstalk will be mostly common mode. Note that this is not using ethernet as Wouter pointed out, only wiring commonly used to implement ethernet. Ethernet itself implies a lot more than just the cable.
Since you have multiple twisted pair in the same cable and these can crosstalk to each other in common mode, you want to send data differentially over each pair. This is also what ethernet does. There are differential line driver and receiver chips out there, which would be a good idea to use if you want to transport individual digital signals from one end to the other. If you have signals that are already inherently differential, like CAN for example, then you use the pairs directly.
Look up what the impedance of CAT5 cable is and try to terminate the ends with that. I know ethernet uses 50 Ω terminations at the receiving end, but I wouldn't just copy that since it may include some other compromises.
One problem of using CAT5 cable with RJ-45 connectors is that it will look like ethernet and someone is going to plug it in to a ethernet port, or plug ethernet into your device. If you are going to use CAT5 cable for something other than ethernet, I recommend to use something other than RJ-45 connectors on the ends.