Raw RGB from CMOS image sensor to grayscale conversion

I am trying to produce an image with OV7740 color image sensor. As far as I know, the sensor is configured by default to provide 10bit parallel RAW RGB data (due to an on-chip Bayer filter). How should this data be processed/scaled in order to obtain a grayscale image data?

EDIT: I guess, the discussion of my issue can be found here.

• what did you try so far? What kind of board are you working with? There are a lot of registers in those sensors... Can you get that datasheet. I know the bigger ones are all under NDA. Finally what are you connecting it to? – Some Hardware Guy Dec 5 '14 at 17:42
• Have you tried using the normal grayscaling algorithm? – Ignacio Vazquez-Abrams Dec 5 '14 at 18:15
• @SomeHardwareGuy I read the digital output from the sensor and display this data as a 10bit grayscale image. I expect the sensor to provide a reasonable output by default (raw rgb, all adjustments are in auto mode), so I did not consider any of the registers. Can not get a datasheet. I drive it with an FPGA and then redirect the data to the NI PCI card for sampling. Thus, the data that I sample must be processed to give some reasonable image. – Nazar Dec 5 '14 at 18:16
• @IgnacioVazquez-Abrams I guess, I could use it (I think you mean scaling 0.21 R + 0.72 G + 0.07 B). However, I first need to convert it from raw rgb to rgb. I was hoping that there is a shortcut to convert raw rgb to grayscale, since the sensors sees everything in graycale. – Nazar Dec 5 '14 at 18:21
• I took a look at the product brief and it claims to be able to output 8-bit YUV. You should be able to gamma-scale the Y channel. – Ignacio Vazquez-Abrams Dec 5 '14 at 18:32

A top quality demosaic process is fraught with subtlety, but very cheap answer can be had if you can accept a 50% loss in spacial resolution. Simply sum groups of four sensor pixels to get a single grey pixel. Assuming your sensor has a "typical" Bayer filter pattern, there will be twice as many green sensors as either red or blue. Summing square groups of four pixels will then make Y = 2*G + R + B, which does over-emphasize the blue component but for simple cases like verifying optical system focus that will not be an issue.

So for a typical arrangement of pixels which looks like this:

G R G R G R....
B G B G B G
G R G R G R
B G B G B G
.
.
.


You would get one Y output for each group of four, at a lower spacial resolution:

Y   Y   Y

Y   Y   Y


The implementation of this can be done in hardware (as in an FPGA) with only enough memory to store half a scan line, as you can sum the input pixels in pairs the first line, then in pairs and including the previous line's sum on each second line.

You could also generate as many Y pixels as inputs by doing a sliding window, that would require a full scan line's worth of memory in hardware.

Doing it "right" involves a number of processing steps, but the core operation is to apply spacial filters to estimate separate R, G, and B monochrome images at identical resolution, with care taken to align them properly. Given those aligned images, it is straightforward to compute Y at each pixel.

An implementation of a fairly sophisticated demosaic system can be found in dcraw, which consumes raw sensor data and emits JPG images. Raw sensor data can be had from many higher end digital cameras, but usually encumbered by trade secrets making its interpretation difficult for third parties. Coffin and his user community have put a great deal of effort into reverse engineering the raw file formats written by such cameras. But it can also be used in your context.

When developing camera projects in the past, I have taken raw pixels straight from CMOS sensors as captured in RAM, dumped them to a file, and passed the file through dcraw.

The related problem for a prototype camera is getting your first prototype optics in focus, in order to have pixels that could mean something and thus prove that the whole thing works by showing a finished JPG of your bench.

The first step is to find and use the test-pattern modes in your sensor. With that, you can usually get reference patterns with known content to test your electronics and data flow. Then you can switch to real optics with some confidence that the raw pixels are meaningful and concentrate on getting the optics to give you an in-focus image.

A technique I've used in the past for a camera based on an M12 mount lens holder was to bring the lens into fairly decent focus with the lens holder mounted on an evaluation board that had software that provided a live view on a PC. Then I locked down the lens, and moved the whole lens holder to my prototype. Due to slight variations in sensor position relative to the PCB and mounting hole it wasn't perfect, but it was good enough to get a recognizable image.

• You are right. I should just do simple math as you said and get my system in focus. Then I will be able to play with it with more confidence. Thank you for your advice. – Nazar Dec 8 '14 at 14:02
• RBerteig, I really like your answer, and I would love to hear more about your project about taking out Raw pixels straight from CMOS sensors as captured in RAM, dumped them to a file, and passed the file through dcraw. I'm trying to understand the function and the hardware to accomplish such task. Thanks a lot, I know it is a old topic. – user111951 Jun 3 '16 at 9:37