What a camera measures

A camera measures two things per pixel: bearing (the direction light came from) and intensity (how much light arrived from that direction). It does not measure range. A single image fixes the direction to every visible surface but not the distance along that direction, so depth has to be recovered by other means (stereo, structured light, or time-of-flight).

This is the complement of lidar. Lidar measures range and bearing and is sparse; a camera measures bearing and intensity and is dense, with an order of magnitude higher angular resolution. A modern sensor delivers millions of bearing samples per frame, which is why cameras perform semantic tasks in perception: lane lines, signs, traffic lights, text, and object class are all best defined by intensity and color, not geometry.

Waymo autonomous vehicle with roof sensor pod — **Figure 1** — A Waymo autonomous vehicle. Cameras sit alongside lidar and radar in the roof and perimeter sensor pods. Source: Dllu, CC BY-SA 4.0, via Wikimedia Commons.

Starship sidewalk delivery robot — **Figure 2** — A sidewalk delivery robot. The band around the body houses the cameras used for obstacle detection and localization. Source: Sillerkiil, CC BY-SA 4.0, via Wikimedia Commons.

The pinhole model

The standard model for a camera is the Pinhole camera model. A 3D point P = (X, Y, Z) in the camera frame projects to pixel (u, v):

u = f x X / Z + c x v = f y Y / Z + c y

The four numbers f_x, f_y, c_x, c_y are the intrinsics: focal lengths in pixels and the principal point. They pack into the camera matrix

K = [f x 0 c x; 0 f y c y; 0 0 1 ]

Pinhole camera projection diagram — **Figure 3** — The pinhole camera model: a 3D object projects through a single aperture to an inverted image on the sensor plane. Source: DrBob and Pbroks13, public domain, via Wikimedia Commons.

Image pixels do not map to a point, each one maps to a ray: the set of all 3D points that land on that pixel (which is why a single camera cannot directly measure range).

Lenses bend these rays, so the pinhole model is paired with a distortion model before it appropriately models a physical camera. The projection and distortion parameters together are recovered by intrinsic calibration, and the math is standardized by the OpenCV camera model.

How robots use cameras

Object detection and semantics: classification of things like lane lines, signs, traffic lights, and free-space, almost always from intensity and color.
Visual odometry and visual SLAM: track features across frames to estimate ego-motion and build a map.
Localization: match the live image against a prior map or known landmarks.
Depth: triangulate across two cameras, or pair a camera with an active illuminator. See Depth.

Downstream autonomy functions depend on modeling each pixel’s ray accurately, which is why lens choice and calibration accuracy matter as much as the sensor.