Are I pictures always the highest-quality pictures?

No. The label describes prediction structure; actual quality depends on encoding choices and source content.

Does a B picture always reference a future picture?

No. Familiar examples do, but some low-delay configurations use only past display-order references.

Can playback order differ from decoding order?

Yes. References must be decoded before dependent pictures, even when displayed later.

AI Glossary / MLOps and deployment / Beginner

Video compression picture types

Definition

Video compression picture types classify pictures by the prediction methods and reference relationships used to encode them. Familiar labels include I pictures, which use intra-picture coding; P pictures, which support predictive coding; and B pictures, which support prediction using two reference lists. Their exact meaning depends on the video coding standard.

What it means

A video contains many pictures that resemble one another. Encoding every picture independently can repeat information. Prediction lets an encoder describe content using already decoded reference information and a coded difference, reducing repetition when useful.

Picture types describe permitted coding relationships, rather than a simple ranking of visual quality. A B picture is not automatically blurry, and an I picture is not automatically lossless. Quantization, bitrate allocation, source complexity and encoder choices also affect the reconstructed result. Furthermore, picture-level labels simplify standards that may specify prediction types at the slice or block level.

How it works

An I picture reconstructs its content without inter-picture prediction. A P picture can use reference pictures, while individual regions can still use intra coding when appropriate. A B picture can combine predictions associated with two reference lists. In familiar arrangements those lists include pictures earlier and later in display order.

The decoder must reconstruct reference pictures before pictures that depend on them. Consequently, decoding order may differ from viewing order. A future reference in display order must already be available when its dependent picture is decoded.

Modern arrangements are more flexible than the introductory pattern suggests. B pictures can themselves serve as references in some configurations, and low-delay B configurations can use past pictures only. In H.264, an IDR picture adds a reference-reset property; an ordinary I picture does not necessarily establish the same clean boundary. These distinctions matter when seeking or joining a stream.

Three practical examples

1. Recorded training video: an encoder uses prediction across nearby pictures to avoid repeatedly describing an unchanged background. Scene changes can require different choices from a steady interview.

2. Interactive demonstration: a team examines whether waiting for future display-order references contributes unacceptable delay. It evaluates a low-delay configuration using measured responsiveness and quality.

3. AI clip extraction: a pipeline requests a short section of compressed video. It may need to decode supporting reference pictures before delivering the first requested image to a model.

Think of it this way

Think of preparing illustrated instructions. One page describes a complete scene, while another says how that scene changes. A third can use two previously prepared illustrations to explain an intermediate position. The preparation order may differ from the order readers see.

The analogy explains dependency and ordering, but actual codecs predict numerical image information and encode differences. They do not understand scenes as an illustrator would, and their reference rules are defined precisely by the format.

Worked example

Consider four displayed pictures: I0, B1, B2 and P3. In this teaching arrangement, P3 references I0, while both B pictures reference I0 and P3.

The display order is I0 → B1 → B2 → P3. A valid decoding order is I0 → P3 → B1 → B2, because P3 must be reconstructed before the dependent B pictures. Display timestamps preserve the intended viewing sequence.

At 30 pictures per second, B1 represents approximately 33.3 milliseconds and P3 represents 100 milliseconds relative to I0. Their separation is approximately 66.7 milliseconds. This illustrates the future-reference horizon for B1 in this arrangement, not a complete measurement of streaming latency. Capture, encoding, transmission, buffering and display contribute additional timing effects.

Illustrative case study

Fictional case: FrameRelay develops a service that extracts demonstration clips for an internal learning assistant. Its prototype begins every requested clip at the nearest picture marked I. Some extracted clips fail to decode correctly in isolation.

The team discovers that its selection rule ignores surrounding reference relationships and decoder initialization. It updates the pipeline to identify suitable random-access boundaries, supply required configuration information and decode any necessary lead-in before trimming the displayed output.

The lesson is that picture labels help explain compression structure but do not replace format-aware extraction. FrameRelay is illustrative; no real company result or universal extraction recipe is implied.

Common misconceptions and mistakes

Do not assume that every I picture resets every dependency. The distinction between intra coding and a suitable random-access point matters.

Do not equate B pictures with mandatory future references in every configuration, or assume they can never become references themselves. Consult the relevant codec and profile.

Do not remove every other compressed picture as though the stream were a folder of independent images. Dependency information, timing and format rules determine whether the result remains decodable.

Business use

Picture-type choices affect compression efficiency, joining behavior, seeking and latency. Teams publishing recorded material may accept different dependency structures from teams supporting interactive communication.

A useful evaluation records the codec configuration, reference structure, access-point spacing and measured playback behavior alongside file size and visual quality. For AI processing, also measure the effort required to obtain particular frames. A smaller encoded file can still require substantial decoding before a requested image becomes available.

Check your understanding

Why can a decoder process P3 before displaying B1 in the worked example?

Answer: B1 depends on P3. The decoder reconstructs P3 first, retains the required reference information, and displays pictures according to their intended timestamps.

Related terms

Video codec
inter-frame prediction
intra-frame prediction
group of pictures
reference picture
decoding order
presentation timestamp
IDR picture
random access

Learn next

Learn about groups of pictures and random-access points next. Then examine one short encoded stream with a media inspection tool, comparing picture labels, decoding timestamps and presentation timestamps. Interpret the results using that codec's documented rules.

Sources

Microsoft, H.264 frame-type enumeration: https://learn.microsoft.com/en-us/windows/win32/api/d3d12video/ne-d3d12video-d3d12_video_encoder_frame_type_h264

Microsoft, Direct3D video encoding specification, reference lists and low-delay configurations: https://microsoft.github.io/DirectX-Specs/d3d/D3D12_Video_Encoding_H264_HEVC.html

IETF, RFC 6184, H.264 terminology and IDR behavior: https://www.rfc-editor.org/rfc/rfc6184.html

Primary documentation consulted. The timing example and fictional case are original illustrations.

Previous: Video codec Next: Video content analysis