Your best bet is to probe the pins with an oscilloscope. If you don't have one or can't afford one then you can use a software scope connected to your sound card. It will be rubbish for reading voltage values but good enough to see digital and PWM signals.
The next best thing is google. I always google before I reverse engineer something to at least know what to look for. Just search for "remote control toy car signal" and one of the top results will lead you to this page: http://electronics.howstuffworks.com/rc-toy2.htm
Basically cheap toy cars work by sending pulses to send a command and the receiver counts the number of pulses to figure out what to do (note that this is a very different signaling scheme compared to "real" RC equipment, those found on more expensive "toys").
The pulses look something like this:
____ ____ _ _ _ _
____| |____| |_| |_| |_| |_| |____
long pulses short pulses
indicates start encode the command
The number of short pulses encode the command. For example, on a toy car I once hacked I found this encoding:
- 10-15 pulses = forward
- 18-24 pulses = reverse
- 26-30 pulses = forward left
- 32-38 pulses = forward right
- 40-45 pulses = reverse left
- 48-54 pulses = reverse right
Last time I used one of these to send my own data I generated the expected pulses using a microcontroller and on the receiving end have a microcontroller read the voltages of the H bridge outputs controlling the drive and steering motors. Then I invented my own coding scheme to send bytes. Something like this:
- forward (F) = start of byte
- forward left (FL) = 00
- forward right (FR) = 01
- reverse left (RL) = 10
- reverse right (RR) = 11
So to send the letter 'A', which is 0x41, I'd transmit:
F, FR, FL, FL, FR
Considering that each "packet" takes around 80ms on average to transmit and you need 5 packets per byte and to be safe and avoid glitching you should send at least 2 or 3 frames per packet, you're looking at roughly 1 second per byte. Which is 8bps. Very slow but usable.
If you don't need to send bytes then just read the motor output directly. That gives you 6 momentary signals (F,R,FL,FR,RL,RR) or 3 on/off signals (F/R, FL/FR, RL/RR).