Color and the image pipeline

A silicon pixel measures intensity, not color. Color comes from a filter mosaic over the pixels and a processing pipeline that reconstructs full color from it. Both steps cost light and add artifacts, which is why some autonomy applications use mono camera images.

The Bayer filter

Most color sensors use a Bayer color filter array filter: a repeating 2×2 mosaic of red, green, green, blue, with twice as many green pixels because human luminance perception peaks in green. Each pixel now sees only one color, so two thirds of the color information at every pixel is missing by construction.

Bayer color filter array mosaic — **Figure 9** — The Bayer color filter array: the RGGB mosaic over the pixel grid, and a cross-section showing one color filter per photosite. Source: Cburnett, CC BY-SA 3.0 and Cburnett, CC BY-SA 3.0, via Wikimedia Commons.

Demosaicing

Demosaicing (debayering) reconstructs full RGB at every pixel by interpolating the missing channels from neighbors. Quality varies:

Bilinear interpolation is fast and produces visible zippering and false color on edges.
Malvar-He-Cutler (gradient-corrected) is the common quality default.
Edge-aware methods interpolate along edges to suppress color fringing.

OpenCV exposes all three through cvtColor (for example COLOR_BayerBG2BGR, the _MHT Malvar-He-Cutler variants, and the _EA edge-aware variants). Demosaicing artifacts land on exactly the high-frequency edges that feature detectors and lane finders care about, so the algorithm choice is not cosmetic.

Demosaicing artifacts on fine detail — **Figure 10** — Demosaicing artifacts (zippering and false color) on fine, high-contrast detail, where interpolating the missing colors is least reliable. Source: Mercury13, CC BY-SA 3.0, via Wikimedia Commons.

Other mosaics exist (CYGM, RGBE, Fujifilm X-Trans, and the layered Foveon sensor), but Bayer dominates and is what almost every robotics color camera uses.

The ISP pipeline

Raw Bayer data becomes a viewable image through an image signal processor, on the sensor or on the host. A typical pipeline: black-level subtraction, white balance, demosaic, color correction, gamma or sRGB encoding, denoise, and sharpening. Each stage is tuned for human viewing, and several are nonlinear and lossy with respect to the original photon counts.

RAW is the sensor data before most of this, linear in light and closest to the physical measurement.
Processed (sRGB) is convenient and compact but has been gamma-encoded, white-balanced, and sharpened, which distorts radiometry.

Monochrome sensors

A monochrome sensor collects roughly two to three times more signal than the same sensor with a Bayer filter, has no demosaicing artifacts, and resolves finer detail because there is no color interpolation.

This is why stereo, visual odometry, and feature tracking frequently run on mono: they want photons and sharp edges. Color sensors are more suitable for perception (classification, segmentation, object tracking).