Output formats and compression

The format in which a camera outputs pixels determines the bandwidth, latency, and computational load placed on downstream systems. The image content is independent of the encoding.

Resolution and frame rate

Resolution is the pixel count (VGA: 640×480, 720p: 1280x720, 1080p: 1920x1080, 4K: 3840x2160); higher resolution carries more angular detail and more data.

Resolution, frame rate, and the resulting data rate are coupled: doubling resolution or frame rate approximately doubles the data rate. Higher frame rate reduces latency and motion blur, while higher resolution increases angular detail and detection range. Interface bandwidth and compute bound the product of the two, so resolution and frame rate trade against each other.

Pixel formats

Before compression, pixels are laid out in a raw format:

RAW (Bayer): sensor data before demosaicing, the smallest representation before an ISP runs.
RGB / BGR (RGB24): three bytes per pixel.
YUV with chroma subsampling: luma (Y) is separated from chroma (U, V), and chroma is stored at lower resolution because human vision is less sensitive to color detail. YUV422 halves chroma horizontally; YUV420, and the NV12 memory layout, halves it in both axes for 12 bits per pixel. Most pipelines and codecs operate on YUV420/NV12.
Y8 (mono): one byte per pixel from a monochrome sensor, the smallest uncompressed format.

Chroma subsampling ratio diagram — **Figure 11** — Chroma subsampling. Luma (Y) is kept at full resolution while chroma is stored more coarsely, here at 4:4:4, 4:2:2, and 4:2:0. Source: Brion VIBBER and Mysid, CC BY-SA 3.0, via Wikimedia Commons.

Compression

Uncompressed: lowest latency and lossless, but largest. A 1080p30 RGB stream is approximately 1.5 Gbit/s.
MJPEG: each frame is compressed independently. Latency is low and constant, and a dropped frame does not corrupt subsequent frames, but compression is weak.
H.264 (AVC): compresses across frames using motion prediction, producing much smaller streams at the cost of encode and decode latency and compute; a corrupted frame propagates until the next keyframe.
H.265 (HEVC): approximately half the bitrate of H.264 at equal quality, with higher compute.
AV1: further bitrate reduction and royalty-free, with the highest compute of the four.

Independent-frame formats (uncompressed, MJPEG) preserve per-frame integrity and exact pixel values. Interframe codecs (H.264, H.265, AV1) reduce bitrate substantially but add encode and decode latency, and their lossy artifacts alter pixel values, which affects photometric and disparity computations.

Interfaces and transport

The output data rate must be carried by the physical interface: USB, MIPI-CSI on embedded carriers, or GMSL/FPD-Link for long automotive cabling (see Depth, synchronization).

In ROS, image_transport abstracts raw, compressed, and Theora streams, allowing a subscriber to select an encoding independently of the publisher.