# How does the Infineon REAL3™ depth sensing imager chips measure time of flight for each pixel?

tl;dr: How do the REAL3™ depth-sensing imager chips measure time of flight of light for each pixel? Is it quadrature, time-to-amplitude, or something else?

The Infineon REAL3™ depth sensing imagers are used to make the Time-of-Flight (ToF) cameras in the upcoming Asus Zenphone as the "depth sensing 3rd rear camera" for Tango implementation. (additional background info)

From the drawing and discussion in the 2 page PDF I believe it uses RF-modulated infrared light for illumination. The modulation signal is also passed directly to the sensor chip serving as a time and/or phase reference.

I'd like to read about how the measurement is actually implemented within each pixel. This is what I've thought so far:

1. I'm assuming these are CMOS photodiodes rather than CCD pixels, so one possibility would be to do some kind of quadrature detection, and simply report phase. The problem here would be the ambiguity if the round-trip time is greater than one period of the modulation. The sensors have dynamically adjustable modulation frequency, which I suppose could be used for "auto-ranging". Also, the quadrature method would require at least approximately sinusoidal intensity variation of the LED.

2. Edge-detection (time to amplitude conversion) could be used instead of quadrature. That would would with rectangular wave modulation.

3. To get rid of super-period time ambiguities, a more complex digital code could be used instead of a square wave.

Considering they've added a microlens array to the imager to improve light collection, the application is probably S/N limited. In the situation of a phone, you're lighting the scene up to 4 meters away with a flashlight running off of your cellphone. To get sub-nanosecond precision with fast update, the battle is noise. It's possible the method with the best inherent S/N performance would be the best choice.

In all of the above, I'm assuming the infrared light is from a single uniform source flood-illuminating the scene, without any grid pattern or structured light.

The highest available modulation frequency is currently 100 MHz. With a period of 10 nanoseconds and the speed of light of about 30cm/ns, that means at the highest frequency objects at a distance of 150cm would be at the ambiguity limit. But dropping to 30 MHz would work for the rated maximum distance of 4 meters.

So far all I've found is the two page PDF linked above, but no data sheet or in-depth discussion.

Can someone explain: how does the Infineon REAL3™ depth sensing imager chips measure time of flight for each pixel? I'm looking for the definitive answer, not idle speculation. However, educated, contemplative speculation is certainly welcome if no answer is forthcoming.

above: screen shot from the 2 page PDF.

• Have you tried hunting in their patents? – pjc50 Feb 23 '17 at 13:43
• @pjc50 also, even if I found an applicable patent, there's no way to know for sure the current chips are using that method. I'm not asking "how could it?", I'm asking "how does it?" – uhoh Feb 23 '17 at 13:55

This paper, Robust 3D Measurement with PMD Sensors , describes in detail the Infineon\PMD tech. sensors and how they work.

Figure 3 shows the individual 'phase pixels' (they call them smart pixels that measure the phase with two photogates. . As the photogates are switched on and off, these collect the light and integrate it. There is a better diagram on wikipedia which also shows the circuit diagram and the simplicity of the sensor pixels, its just a dioded and some charge buckets.

It would be difficult to understand how the sensors could sense with any accuracy because sampling the phase directly would be a challenge. And herein lies the 'secret sauce', what you can't sample directly (the TOF information) you estimate with statistical methods. The paper goes into detail on this and also the equation they use for estimation:

$$d = \frac{c * \varphi}{4\pi f_{mod}}$$

where

$$\varphi = \arctan\frac{A_1-A_3}{A_2-A_4}$$

There is another good presentation here about a different camera that works on the same technology.

Now for the speculation, I don't think they are doing too much beyond 4 phase sampling in their sensors. They may be comparing the depth and phase information between pixels (more than just a 2d smoothing filter) which would give you a little more resolution.

• OK, this is very helpful! Figure 3 speaks a lot by itself - an actual cross-section of the pixel device showing the charge division controlled by relative time between optical and reference signals. I found Reference 6 here. – uhoh Feb 28 '17 at 23:57
• I think the standard deviation "figure 6" in the paper that shows their theoretical value with the measured value is really cool. The tech is really cool too, its an easy way to estimate distance without using a TOF circuit that you'd need to calibrate on every pixel. – Voltage Spike Mar 1 '17 at 0:05
• I'm kind-of hinting to you that including Figure 3 (in addition to, or instead of Figure 4) would make this an even better answer from my point of view. – uhoh Mar 1 '17 at 0:06
• Its a good idea, figure 4 looks bad unless you clock on it and the wiki diagram looks better. – Voltage Spike Mar 1 '17 at 0:20

It uses a Photonic Mixer Device.

The REAL3 imager uses a sensor from pmd technologies(see the bottom of that page). PMD technologies is named for their use of Photonic Mixer Device technology.

As you speculated, it does use pulsed light. Specifically, it modulates the outgoing beam with an RF carrier and then measures the phase shift of the return signal. It is captured with a CMOS sensor.

If you want to dig deeper into how phase is captured from a CMOS pixel, PDM technologies only states that the CMOS is "coupled" with a DSP. But there are academic papers explaining the process such as this one (to be fair, this is from canesta, a competitor of PMD). One of the first questions that came to mind for me is how they would capture timing data, a typical CMOS is coupled with a shift register, and polling pixels by reading the rows would probably not be fast enough to capture an RF frequency. Even at the lowest RF frequency of 300 kHz, polling would require you to access each pixel in the GHz range to get any meaningful phase comparison data.

It turns out it modulates the gates for the pixel sensors in sync with the light, then it reads the voltage difference between the two nodes.

To quote the paper linked to above "The differential structure accumulates photogenerated charges in two collection nodes using two modulated gates. The gate modulation signals are synchronized with the light source, and hence depending on the phase of incoming light, one node collects more charges than the other. At the end of integration, the voltage difference between the two nodes is read out as a measure of the phase of the reflected light. Thus, in effect, this pixel simultaneously performs the function of mixing and low pass filtering."

Finally here is the relevant patent.

• These links are all very helpful! Figure 5 and reference 9 in Canesta seem to explain the principle of the semiconductor per-pixel device itself (charge division based on timing). – uhoh Mar 1 '17 at 0:01