0
\$\begingroup\$

You all probably know how a CMOS sensor works, at least in general. In short, it consists of a lot of pixels (photodiode plus amplifier and so on), and each pixel is read out individually (different to CCD).

I have two questions now, a main and a minor one:

The minor: A pixel generates a signal only when the signal is above a certain threshold. Physically speaking, this is when a certain amount of light particles hit the pixel aka the photodiode. So, when it is dark, the pixel is not generating any signal (roughly phrased), right?

The main one: How could a CMOS sensor be adjusted such that a pixel is read out only when a change in the signal is significant? With the help of a simple resistor, one could change the threshold for the necessary light intensity but this would ignore a signal change from, e.g., very bright to very dark. Would it already work with a capacitor? It would be charged when it is bright and when it changes to dark, the capacitor would be discharged. And when it is bright again, the capacitor would be charged again.

Here are two images to help with visualization. Though one cannot ignore physics, I'm more interested in the electronics/engineering part.

img1

But here you can already see even three transistors and, for me, it seems like T_1 is already doing this job?

\$\endgroup\$
10
  • \$\begingroup\$ Well, now as I read my own question, I think each CMOS sensor is actually only detecting changes, or? No light -> no signal. Light -> signal. So a change in both directions causes a straight clear binary signal..?! \$\endgroup\$
    – Ben
    Dec 1, 2021 at 9:39
  • \$\begingroup\$ I suggest that you take a step back and consider what happens to the voltage across the photodiode when: 1) the cell is in reset so T1 conducts 2) now T1 opens but no light falls on the diode. 3) reset again (T1 closes then opens) now light does fall on the diode. It might help to consider that the photodiode behaves as a small capacitor which is sensitive to light, when light hits the diode, a small current can flow between the anode and cathode, as if the diode leaks. \$\endgroup\$ Dec 1, 2021 at 10:06
  • \$\begingroup\$ To make a sensor that responds to changes only, the simplest solution is to read out the complete sensor (grab the image) TWICE and then (numerically) compare the difference. \$\endgroup\$ Dec 1, 2021 at 10:09
  • 1
    \$\begingroup\$ You would have to read the data out anyway to detect if it has changed. The CCD just provides image data, next level up is processing it, e.g. comparing difference to previous image. \$\endgroup\$
    – Justme
    Dec 1, 2021 at 10:51
  • 1
    \$\begingroup\$ When T_1 is controlled externally anyway, how could I read out a pixel immediately when there is a change? T1 is controlled by the image sensor, you have no direct control over it. This type of image sensor isn't designed for your "detecting changes", for that a different setup is needed, see DamienD's answer. \$\endgroup\$ Dec 1, 2021 at 12:02

2 Answers 2

4
\$\begingroup\$

There has been a lot of work (research and commercial) done on this subject, so you will be able to find answers in the literature.

What you're describing is generally known as an event camera. The advantages compared to reading out whole frames and computing changes digitally are the following:

  • vastly increased temporal resolution and dynamic range
  • low power consumption
  • sparse (low-bandwidth) output.

In an event camera each pixel independently tracks changes in light intensity with a differencing circuit. Whenever the change is greater than a certain threshold the comparators in the pixel emits an event, which is sent out as the pixel address, timestamp and sometimes the polarity of the change.

You may want to have a look at one of the seminal papers for details of a possible implementation in a custom CMOS sensor:

enter image description here

P. Lichtsteiner, C. Posch and T. Delbruck, "A 128 x 128 120 dB 15 μs Latency Asynchronous Temporal Contrast Vision Sensor," in IEEE Journal of Solid-State Circuits, vol. 43, no. 2, pp. 566-576, Feb. 2008, doi: 10.1109/JSSC.2007.914337.

https://www.researchgate.net/publication/2983746_A_128_128_120_dB_15_ms_Latency_Asynchronous_Temporal_Contrast_Vision_Sensor

\$\endgroup\$
3
  • \$\begingroup\$ Thanks a lot! Didn't expect it to be such an effort. Thanks for the information, I will dive deeper into it! \$\endgroup\$
    – Ben
    Dec 1, 2021 at 11:33
  • \$\begingroup\$ I was reading the Wikipedia article for event cameras and there it is given "Event cameras do not capture images using a shutter as conventional cameras do. Instead, each pixel inside an event camera operates independently and asynchronously, reporting changes in brightness as they occur, and staying silent otherwise." How is that? Isn't also every pixel in CMOS and even CCD independent from each other? I mean the readout is different and, more or less, synchronized in the case of CMOS and CCD but the pixels (the photodiodes) work independently from the readout, or not? \$\endgroup\$
    – Ben
    Dec 6, 2021 at 9:01
  • 1
    \$\begingroup\$ @Ben you can think about it this way: in a normal frame camera the interface chip reads out the pixels. In an event camera it's the other way around -- each pixel contains the circuit to detect changes and signals the event to the interface. \$\endgroup\$ Dec 6, 2021 at 11:19
1
\$\begingroup\$

The readout is driven from outside the sensor. You have to scan the rows and columns and read each pixel individually. Until you have read it, you can't tell if it has changed.

Every pixel will have some output voltage, regardless of light level. The pixel doesn't detect light above some value and only then "turn on."

The output is an analog voltage that varies from a low value (in the dark) to a high value (when light shines on it.)

The diode itself acts to some extent as a capacitor. It integrates the light intensity to an average over time.

You hit RST to reset the pixel (minimum output voltage,) wait a short time for light exposure to charge the diode's capacitance, then read the output voltage.

Detecting a change in light intensity means reading the same pixel twice then taking the difference of the two readings.

\$\endgroup\$
3
  • \$\begingroup\$ Thanks, so each pixel is read out individually, as I mentioned? What has to be changed when a pixel is only and "immediately" read out when it detects changes in the light intensity? \$\endgroup\$
    – Ben
    Dec 1, 2021 at 10:28
  • 1
    \$\begingroup\$ Throw the whole thing away and start from scratch - or read twice and take the difference as stated. CMOS image sensors cannot signal changes only. \$\endgroup\$
    – JRE
    Dec 1, 2021 at 10:31
  • \$\begingroup\$ Not in any case?? I mean by software/chip it is obviously possible though it wouldn't be the CMOS sensor, then. But it doesn't appear so special to me that it has to be impossible. \$\endgroup\$
    – Ben
    Dec 1, 2021 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.