I was wondering how does short-range radar and LIDAR work? Considering the speed of light, the receiver sampling rate would need to be extraordinarily high to detect short-range distances(sub 1-2 feet). How do detectors work that avoid needing this high sampling rate? Is there some other way to measure distance than flight time counted by digital ticks?
I don't think this site is a place for comprehensive technology reviews or dissertations on microwave-based technology. In short, light travels in air at 12 inches per 1 ns. So this is not completely out of touch for modern electronics to detect sub-ns delays.
Here is one device, after a 30-second search, TI AWR1443 single-chip radar. So the answer is that radar processors avoid ultra-fast sampling rates by "baseband processing" (aka "quadrature demodulation") of the radar carrier signal (70-90 GHz range) into more manageable frequency range, and then uses a 200-MHz ARM processor to get the final result.
Is there some other way to measure distance than flight time counted by digital ticks?
Yes indeed. If you look at Ali Chen's block diagram, you'll see an example of using a chirp waveform.
The transmit beam uses a linearly increasing (or decreasing, it doesn't matter) frequency, which is also applied to the input mixers. Notice the block labelled "ramp generator"? Since the transmit frequency is applied to the input mixers, the output of the mixers is the difference between the input and output frequencies.
Let's say the output varies at 10 GHz/second. Then, at a range of 1 foot, since the receive signal is delayed by 2 nsec, the input will be offset from the output by 20 Hz, and the ADC will have no trouble handling this.
You'll note that this doesn't allow fast range acquisition. The range determination will probably take a noticeable fraction of a second, particularly for close objects, but for automotive use that's not a big problem.
Another approach is phase discrimination. Using a constant-frequency transmitter (no chirp) if you compare the input and output you can get the phase difference (assuming you're within the coherence length of the transmitter waveform) between the two. Knowing the wavelength allows determination of the distance. This works well with gas lasers in LIDAR, for instance, but it's not commonly found in semiconductor lasers, since they usually have rather short coherence lengths.
This also has drawbacks, the biggest being range ambiguity. If you get, for instance, 360 degrees of phase shift at 10 feet, you can't tell the difference between 11 feet and 1 foot. This is sometimes not an impossible problem, since receive intensity will drop off rapidly with increasing range, but it is a real consideration.
For autonomous driving applications in particular (what the current hype is all about), there are two main technological avenues being considered.
Time of Flight (ToF) This is the one that uses light pulses and involves the speed of light rather directly. You're completely right that acquisition is very challenging for short distances, requiring both high speed and low electronic jitter. The usual distances involved are a meter to several meters.
Frequency Modulated Continuous Wave (FMCW) You take a continuous wave (single frequency) laser, and modulate its carrier frequency with some specific waveform, often a linear chirp. You send this modulated light wave in the environment and collect it back after some delay. You mix this delayed light wave with a non-delayed version of what you sent. These two light waves are now at slightly different frequencies because of the time-dependent modulation. The process of mixing them on a photodetector is called optical heterodyne detection, and is completely analogous to the RF concept of the same name. As it turns out, photodetector are ideal mixers, as they respond quadratically to the electric field – or linearly with respect to power, itself proportional to the electric field squared. This mixing creates a beat note at the frequency difference, giving you the distance. Admittedly, resolving smaller distance requires very fast modulation, which is ultimately the limiting factor.
Another method of remote sensing, which is only viable for certain specific applications (most often biological) is Optical Coherence Tomography (OCT). Many variations exist under this name, that are ultimately equivalent to each other. One of them is to sweep the wavelength of a laser and make the back-reflected light interfere with a fixed reference. By taking the inverse Fourier transform of the resulting spectrum, you are able to extract the time-domain response of the system, up to a time/length offset.
Older LIDAR such as from ERIM Environmental Research Institute of Michigan, used in Terragator projects at CMU, used 10MHz-modulated laser light. As whatroughbeast explained, there will be ambiquity-resolution tasks.
Modern LIDAR use pulses; Velodyne specs suggest 5 picosecond resolution (distance is unspecified) yet a pulse width of 5 nanoseconds. All these pulses may be ambiguous (Velodyne uses dozens of lasers, all presumably timed to avoid ambiguity).
The human-eye and human-brain cooperate to produce a 3_D model of our surroundings, that 3_D model updates many times a second as
1) moving edges occur where not expected in the 3_D predictions
2) something moves, where a constant (not-threatening) visual field was the initial hypothesis
The ability to foveate, to bring very high resolution to some puzzling or newly threatening feature in the image, is just now being implemented in some LIDAR.
Humans generate massive information flows in our eyes, and filter out almost all of the changes/information.
LIDAR thus far assumes, being an orthogonal sensor modality to standard cameras or RADAR, some sloppy resolution will be the cure for autonomous-vehicle flaws.
This assumption is not panning out, because the information-fusion is not a solved problem, per CMU folk.
There are a bunch of analog techniques to avoid stupid clock rates.
Laser ranging and 10 digit frequency counters used analog sample-and-hold of a triangle wave to interpolate between clock edges adding 2-3 extra digits resolution to the clock rate
As long as you have analog noise (time jitter), then you could just average lots of readings, to get fine resolution, which is probably what these guys do to time-of-flight over less than a metre.
Note that these are really single target systems.
When you use FMCW / chirped radar you can do an FFT on your (baseband) signal and separate out the different targets which are at different distances.
Just for interest, you should look at how they radar map mars and venus using Arecibo. How do you get this, when the beam is bigger than the planet?