IM Engineer Study Notes
← Back to list
Computing & Embedded
#영상코덱#비디오압축#HEVC#AV1#엔트로피부호화
Last updated · 2026-11-06

Video Codecs and Video Compression (H.264·HEVC·AV1·VVC)

1. Overview

A. Definition

A video codec is a collective term for the algorithms and standards that compress digital video through encoding to cut storage and transmission costs and restore it through decoding on playback. It is a lossy-compression technology that mathematically removes the redundancy of the video signal and exploits the limits of human vision to reduce data volume by tens to hundreds of times while preserving perceptual quality.

The word "codec" is itself a blend of coder and decoder. Video is a huge three-dimensional (width × height × time) signal in which 24–60 frames arrive each second, each frame carries millions of pixels, and each pixel carries color. For example, representing 4K (3840×2160) video uncompressed at 60 frames per second and 24 bits per pixel reaches about 11.9 Gbps (3840×2160×60×24), which no Internet line can transmit in real time. A video codec filters out information that "humans barely perceive" from this raw signal and efficiently encodes only the essential remainder, pulling the same perceived quality down to tens of Mbps or less. In other words, a codec is an engineering product combining a perceptual model of "what can be discarded without humans noticing" and the information theory of "how to represent what remains in the shortest form."

B. Background and Necessity

The explosion of video traffic is the fundamental driver of codec advancement. More than half of global Internet traffic is video, and OTT streaming, video conferencing, CCTV, short-form, and cloud gaming all depend on video transmission. On top of this, resolution has kept expanding from FHD to 4K·8K, frame rate from 30fps to 120fps, and tonal expression from SDR to HDR. Raw data volume grows in proportion to the square of resolution and to frame rate, but the growth rates of network bandwidth and storage media cannot keep pace. The only means of bridging this gap is the continuous improvement of compression efficiency, which is why roughly every ten years a new standard has emerged targeting "the same quality at half the bitrate of the previous generation."

The economic motive is also clear. For hyperscale operators such as Netflix and YouTube, a 1% bitrate reduction translates directly into enormous CDN·bandwidth savings and reduced buffering. Conversely, for users it means higher quality on the same plan·line and, on mobile, viewing while reducing data consumption. In domains with fixed allocated bandwidth such as broadcast·satellite, a more efficient codec means more channels or higher quality. Thus codec efficiency is not a mere technical metric but a key variable that simultaneously governs a service's cost structure and user experience.

The possibility of compression fundamentally arises from the fact that the entropy (uncertainty) of a video signal is low. In natural video, adjacent pixels and consecutive frames are strongly correlated, so the actual information content is far smaller than the raw bit count. Information theory's Shannon limit stipulates a lower bound that "losslessly you cannot reduce below the entropy," but video permits lossy compression that discards the parts humans cannot perceive, so bitrates well below that limit are possible. Codec design ultimately is engineering optimization that "minimizes perceptual distortion within a given bit budget," and this is the theoretical background of the rate-distortion optimization explained later.

C. Key Characteristics

A video codec's character can be summarized in four points. First, asymmetric computation — encoding searches for the optimal prediction mode and motion vector, so it is far heavier than decoding (hardware acceleration is essential for real-time delivery). Second, the rate-distortion trade-off — the fewer bits used, the greater the distortion, so every design reduces to finding the optimal point on this curve. Third, the separation of standard and implementation — the standard specifies only "how the decoder interprets (the bitstream syntax)," while encoding quality is left to the implementation (encoder) to induce competition. Fourth, license dependency — the patent-royalty structure governs adoption rates, and this became the direct backdrop for the emergence of the royalty-free codec (AV1).

These four characteristics are intertwined and determine a codec's real success or failure. No matter how excellent the rate-distortion efficiency, if there is no hardware decoder owing to the asymmetric computation burden, it is shunned on mobile, and if the license is opaque, the ecosystem does not move. Therefore, when evaluating a codec, a perspective that views it comprehensively across the four dimensions of efficiency·complexity·license·ecosystem, rather than the single axis of compression ratio, is especially important from a Professional Engineer's standpoint.

2. Basic Principles of Video Compression and the Hybrid Codec Structure

A. The Four Redundancy-Removal Principles

Video compression removes "information that may be discarded" by dividing it into four kinds of redundancy. Spatial redundancy is the property that adjacent pixels within one frame are generally similar, and it is removed by intra prediction and transform coding. Temporal redundancy is the property that consecutive frames are nearly identical and only part moves; it is removed by motion-compensated prediction — this is the core by which video compression obtains a far higher compression ratio than still images (JPEG). Statistical redundancy is the property that certain symbols appear frequently, and it is removed by entropy coding. Psycho-visual redundancy is the perceptual characteristic that the human eye is sensitive to brightness but insensitive to chrominance·high-frequency detail, and it is exploited by chroma subsampling (4:2:0) and quantization.

These four principles are not independent but are combined into a pipeline. Prediction removes temporal·spatial redundancy, leaving only the residual (the difference between prediction and actual); the residual is transformed·quantized to remove psycho-visual redundancy; and finally entropy coding removes statistical redundancy. The essence of codec evolution across generations has been to make each of these stages "more sophisticated but more expensive" to raise compression efficiency.

flowchart LR
  IN["Input frame"] --> PRED["Prediction<br/>(intra/inter)"]
  PRED --> RES["Residual signal<br/>(original - prediction)"]
  RES --> T["Transform(DCT/DST)"]
  T --> Q["Quantization"]
  Q --> ENT["Entropy coding<br/>(CABAC etc.)"]
  ENT --> BS["Compressed bitstream"]
  Q --> IQ["Inverse quant·inverse transform"]
  IQ --> REC["Reconstructed frame"]
  REC --> ILF["In-loop filter<br/>(deblocking/SAO/ALF)"]
  ILF --> DPB["Decoded picture buffer<br/>(reference frames)"]
  DPB --> PRED

B. The Hybrid Block-Based Coding Structure

Since the 1990s almost all commercial codecs (H.26x, MPEG, AV1) have shared the same skeleton called hybrid block-based coding. "Hybrid" means it combines prediction (predictive coding) and transform (transform coding), and "block-based" means it divides a frame into rectangular blocks and processes it block by block. A frame is first split into large coding units and then recursively broken into smaller pieces according to image characteristics. H.264 uses 16×16 macroblocks (MB), HEVC splits up to 64×64 CTUs (Coding Tree Units) by quadtree, and VVC splits up to 128×128 blocks more flexibly by quadtree + multi-type tree (QTMT). Dividing complex regions finely and flat regions coarsely to allocate bits efficiently is the core of the evolution of the partition structure.

For each block the encoder weighs numerous choices (prediction mode, split depth, transform size, reference frame) and picks the optimal combination by rate-distortion optimization (RDO). RDO is the process of finding the choice that minimizes the cost function J = D + λ·R (D is distortion, R is bit count, λ is the Lagrange multiplier); as codec generations rise, the number of choices explodes and encoder computation grows exponentially. It is precisely this "more tools and deeper search" that is the fundamental reason encoding complexity rises several-fold to tens-of-fold as the price of improved compression efficiency.

flowchart TB
  F["One frame"] --> SPLIT["Block partition<br/>(CTU/CU quadtree)"]
  SPLIT --> DEC{"Block type decision<br/>(RDO)"}
  DEC -->|intra| INTRA["Intra prediction<br/>(directional modes)"]
  DEC -->|inter| INTER["Inter prediction<br/>(motion estimation·compensation)"]
  INTRA --> R["Residual generation"]
  INTER --> R
  R --> TQ["Transform·quantization"]
  TQ --> E["Entropy coding"]
  INTER -.reference.-> REFBUF["Reference frame buffer"]

C. Color Space and Chroma Subsampling

Psycho-visual redundancy is first exploited in color representation. The codec treats the input not as RGB but as the YCbCr color space separated into luma (Y) and chroma (Cb, Cr). The human eye is very sensitive to brightness (Y) but insensitive to the spatial resolution of color (CbCr), so even applying chroma subsampling that halves the resolution of the chroma components causes almost no perceived degradation. The most widely used 4:2:0 halves chroma both horizontally and vertically, compressing color information to 1/4, and this alone substantially reduces data volume before encoding.

Broadcast·general streaming mostly uses 4:2:0, but editing·mastering workflows or screen sharing where the color edges of subtitles·graphics matter use 4:2:2 or 4:4:4 (no subsampling) to prevent color bleeding. That is, the subsampling choice is a trade-off between "how far to exploit the eye's insensitivity" and "how important color precision is for the use case," and the supported range differs by profile even for the same codec. This stage is the codec pipeline's implicit first gateway, already granting large compression gains before prediction·transform.

D. Frame Types and GOP Structure

Temporal-redundancy removal is performed by dividing frames into three types. I-frames (Intra) are self-compressed without referencing other frames and serve as the reference point for random access·error recovery but are large in size. P-frames (Predictive) reference a preceding frame one-directionally and encode only differences, and B-frames (Bi-directional) reference both past and future frames bi-directionally to obtain the highest compression ratio. Their repeating unit is the GOP (Group of Pictures), arranged for example in an I B B P B B P … structure. Lengthening the GOP improves compression efficiency but reduces random-access points and raises the risk of error propagation, so streaming compromises by placing an I-frame at each segment boundary. In practice, the adaptive streaming (ABR) of Netflix·YouTube aligns the start of 2–6 second segments to I-frames so that there is no stutter when quality switches.

3. Core Encoding Techniques

A. Prediction: Intra and Inter

Intra prediction estimates the current block from already-reconstructed neighboring pixels within the same frame. Whereas H.264 offered 9 directional modes, HEVC offers 35, and VVC introduced 67 angular modes plus learning-type tools such as matrix-based intra prediction (MIP). The more directional modes, the more accurately the direction of edges·textures is followed to reduce the residual, but the mode-search cost also rises.

Inter prediction is the heart of video compression. It finds the region most similar to the current block in a reference frame (motion estimation), expresses that positional difference as a motion vector (MV), takes the reference block as the predicted value (motion compensation), and encodes only the residual. As generations rise, MV precision went from 1/4-pel to VVC's 1/16-pel, the number of reference frames and block sizes became more flexible, and advanced tools were added such as HEVC's Merge mode and VVC's affine motion compensation (expressing even scaling·rotation) and bi-directional optical flow (BDOF). For example, in a scene where the camera zooms·rotates, the former translational MV mispredicted, but the affine model approximates even block deformation to greatly reduce the residual — this is one factor behind VVC's marked gain over HEVC on certain content.

Motion estimation is the heaviest stage, accounting for most of the encoder's computation. Theoretically a full search examining every candidate position in the reference frame is optimal but unrealistically expensive, so real encoders reduce candidates with fast-search heuristics such as diamond·hexagon search. They also use a two-stage structure that roughly finds at integer-pixel granularity and then refines to sub-pel granularity through interpolation. Raising sub-pel precision accurately compensates even small motions to reduce the residual, but increases interpolation-filter computation and MV signaling cost. MV prediction (MVP), which takes neighboring blocks' MVs as the predicted value and encodes only the difference, is also an important optimization that removes the statistical redundancy of the MV itself. In the end, the quality of inter prediction is proportional to "how widely and precisely one searches," and this search depth becomes the key knob that divides encoding speed and quality.

B. Transform and Quantization

The residual remaining after prediction is moved to the frequency domain by the Discrete Cosine Transform (DCT) (DST for some blocks). The energy of natural video concentrates at low frequencies, so after the transform it divides into a few large coefficients and many small ones. Next, quantization divides each coefficient by the quantization step (QP) and rounds, and at this stage information is actually discarded (the source of lossy compression). Coarsely quantizing high-frequency coefficients saves many bits while humans barely notice — this is exactly where psycho-visual redundancy removal operates concretely. Raising QP lowers bitrate but increases block noise·blur, so the encoder adaptively adjusts QP per scene (adaptive QP) to preserve perceived quality. VVC further raised transform efficiency with a 64-point transform for large blocks, Multiple Transform Selection (MTS), and Low-Frequency Non-Separable Transform (LFNST).

C. Entropy Coding

Quantized coefficients·MVs·mode information are finally compressed by entropy coding that removes statistical redundancy. H.264 selectively offered CAVLC (variable length) and CABAC (Context-Adaptive Binary Arithmetic Coding), and HEVC·VVC unified on the higher-efficiency CABAC. CABAC updates each symbol's probability of occurrence in real time according to context and compresses by arithmetic coding to less than one bit per symbol, but as the price of this high efficiency, inter-bit dependency grows and parallelization becomes difficult. AV1 uses a similar multi-symbol arithmetic coding. Entropy coding bears a substantial portion of the total gain but is inherently sequential, so it sometimes becomes the decoder bottleneck in high-resolution real-time processing.

D. In-loop Filter

Block-unit processing inevitably produces block-boundary discontinuities (blocking artifacts) and ringing. To mitigate these, a deblocking filter is applied to the reconstructed frame, HEVC additionally layers on SAO (Sample Adaptive Offset), and VVC adds ALF (Adaptive Loop Filter). The key point is that this filtering is done inside the coding loop so that the filtered frame is used as the reference for the next frame. Therefore the encoder and decoder share the exact same reconstruction result and no drift occurs.

The more sophisticated the filter, the higher the reference quality, creating a virtuous cycle in which later prediction improves, but the decoder's computation burden also rises. In particular ALF adaptively derives Wiener-filter coefficients per frame to bring the reconstructed signal closer to the original, and because these coefficients themselves must be carried in the bitstream, the trade-off of "filter gain versus signaling cost" arises once more. Thus every new tool of a codec, without exception, has its adoption decided by weighing the compression gain obtained against the added computation·signaling cost.

E. Rate Control: CBR·VBR·CRF

Even with the same codec·same tools, how bits are distributed along the time axis (rate control) governs final quality and transmission suitability. CBR (Constant Bitrate) keeps the per-second bit volume constant, suitable for broadcast·real-time delivery with fixed bandwidth, but has the drawback of quality collapsing in complex scenes. VBR (Variable Bitrate) spends more bits on complex scenes and saves on simple ones to raise average quality, favorable for storage·VOD. CRF (Constant Rate Factor) fixes a "target quality" rather than a bitrate and varies the bits accordingly, widely used in the x264·x265·AV1 encoders.

In practice these are combined with 2-pass encoding (first pass analyzes complexity, second allocates bits) or VBV (Video Buffering Verifier) constraints. For example, OTT commonly adopts capped VBR, which sets a per-segment peak-bitrate ceiling to prevent decoder-buffer underrun while flexibly allocating bits according to scene complexity. In the end, rate control is the problem of "how to distribute the optimal point of the rate-distortion curve along the time axis within the real constraints of network·buffer," and it directly affects perceived quality no less than codec efficiency.

F. Quality Metrics: PSNR·SSIM·VMAF

To judge that "compression was done well," distortion must be measured objectively. Traditionally, PSNR (Peak Signal-to-Noise Ratio) computes the pixel difference between original and reconstruction and is simple, but its correlation with human perception is low, so cases of "good numbers but bad to the eye" are frequent. SSIM (Structural Similarity) compares brightness·contrast·structure together and is closer to perception, and VMAF (Video Multimethod Assessment Fusion) released by Netflix fuses multiple quality features by machine learning to show high correlation with human evaluation (MOS).

These metrics are used directly as the distortion term (D) of rate-distortion optimization and as the quality target in encoding-ladder design. For example, per-title encoding sets per-content bitrates in the manner of "achieving VMAF 93 at the minimum bitrate." However, no single metric is perfect, so in practice multiple metrics and subjective evaluation (golden-eye) are run together, and dependence on the metric must be stated even in cross-generation efficiency comparisons to be trustworthy. The refinement of quality measurement itself is another lever that "raises quality without changing the codec."

4. Cross-Generation Codec Comparison and Selection

Codec history can be summarized as the trend of "efficiency doubling roughly every ten years." H.264/AVC (2003) popularized Internet video, and H.265/HEVC (2013) opened the 4K era, achieving about 50% bitrate reduction versus H.264. AV1, released in 2018 by the Alliance for Open Media (AOMedia), professed royalty-free and offered higher efficiency than VP9·HEVC, and H.266/VVC, standardized in 2020 by JVET (ITU-T·MPEG jointly), was designed to target again about 50% reduction versus HEVC. However, cross-generation efficiency figures vary widely by resolution·content·encoder maturity, so they are hard to assert as a single percentage, and generally the gain of a newer-generation codec tends to grow at higher resolutions.

The lineage of the royalty-free camp is also important. VP8·VP9, released based on the On2 technology acquired by Google, established themselves on the web·YouTube as alternatives to avoid HEVC's patent burden, and this flow converged into AV1 of AOMedia, formed by Google·Amazon·Microsoft·Netflix·Intel and others. That is, AV1 is not the product of a single company but was born as the industry's collective response to HEVC's license uncertainty. Only by understanding this background can the paradox of "why a codec that is not the most efficient spreads faster" be explained.

The important branch point here is the gap between technical efficiency and adoption. VVC has the highest efficiency, but owing to the experience of HEVC's complicated patent pools, license uncertainty is cited as the biggest obstacle to adoption, and hardware-decoding support is also spreading slowly (e.g., Intel Lunar Lake beginning to support native VVC decoding for PCs). Conversely AV1, though its efficiency falls somewhat short of VVC or is at a similar level depending on content, is pulling ahead in actual adoption owing to the decisive advantage of being royalty-free, large-scale adoption by platforms such as YouTube·Netflix, and the spread of hardware decoding in major browsers·mobile SoCs. In short, HEVC's slow spread and AV1's rise simultaneously show the lesson that not "the best codec" but "the codec equipped with license·ecosystem·hardware" wins.

Codec Standardized (year) Body Efficiency (vs. previous) License Notes
H.264/AVC 2003 ITU-T/MPEG baseline royalty widest compatibility
H.265/HEVC 2013 ITU-T/MPEG ~50%↓ complex patent pools 4K broadcast·UHD
VP9 2013 Google similar to HEVC royalty-free YouTube
AV1 2018 AOMedia exceeds VP9/HEVC royalty-free OTT·web spread
H.266/VVC 2020 JVET ~50%↓ vs HEVC uncertainty persists high efficiency, early adoption

Codec selection is a decision that comprehensively weighs not mere efficiency but the target device's hardware decoding, license cost, latency (real-time) requirements, and ecosystem maturity. Software decoding grows more power·heat intensive the newer the codec, disadvantageous on mobile, so in practice the common strategy is multi-codec encoding: pick "the most efficient codec that is hardware-accelerated" and also provide an H.264 fallback for unsupported devices.

To concretely sense the efficiency difference by way of example, the 4K60 raw video of about 11.9 Gbps computed earlier is compressed by H.264 to roughly 20–30 Mbps, obtaining a compression ratio of about 400–600×. HEVC can deliver the same perceived quality at roughly half that, 10–15 Mbps, and the VVC·AV1 family can deliver at lower bitrates depending on circumstances (actual figures depend heavily on content·encoder settings, so they should be understood as ranges). Because this saving margin converts into CDN cost and buffering rate at the scale of hundreds of millions of viewers, large operators invest in newer-generation codecs and content-adaptive encoding even while absorbing enormous encoding computation. Conversely, small services often remain H.264-centric considering encoding cost·compatibility, so even for the same technology, the optimal choice differs by business scale.

5. Deep Dive: Latest Trends and Encoding Advancement

The evolution of hardware encoding cannot be left out. To go beyond the power·computation limits of software encoding, hyperscale platforms deploy at large scale dedicated encoding ASICs (e.g., YouTube's Argos VCU, dedicated transcoding accelerators) and GPU encoders to economically convert hundreds of millions of hours of uploads to newer-generation codecs. Accelerators excel in throughput·power efficiency but have the trade-off of somewhat lower compression efficiency than software, so a tiered strategy is used by content popularity: "re-process only frequently watched video with slow, sophisticated software encoding." This is a representative case showing that codec selection cannot be separated from encoding infrastructure design.

Apart from generational replacement of the codec itself, the intelligentization of the encoding method is greatly raising real quality. The content-adaptive encoding (per-title / CAE) led by Netflix, instead of using a fixed bitrate ladder for all content, allocates low bitrates to easy-to-compress content such as animation and high bitrates to complex action, greatly saving bandwidth at the same perceived quality. Further, shot-based encoding that optimizes per shot combines with optimization based on perceptual-quality metrics such as VMAF, obtaining substantial efficiency improvement "without changing the codec."

A concept commonly confused here is the distinction between codec and container. A codec is the specification of "how to compress the video bitstream," while containers such as MP4·MKV·WebM are the specification of "how to pack compressed video·audio·subtitle tracks and sync information into one file." Since the same AV1 bitstream can be packed into both MP4 and WebM, playback-compatibility problems must be diagnosed by distinguishing codec non-support from container non-support. Also, HDR (High Dynamic Range) carries color metadata (transfer functions PQ/HLG, color gamut BT.2020) in the bitstream apart from the codec, and HDR10·Dolby Vision·HDR10+ are differences in this metadata system, not the codec itself. Therefore a combination such as "4K HDR AV1" plays correctly only when codec·container·HDR metadata·color space are each compatible.

The most notable trend is AI-based (neural) video compression. Research is active on compressing video into a latent representation with an autoencoder, or replacing specific stages of the traditional codec (prediction·filter) with neural networks. MPEG is pushing NNVC (Neural Network-based Video Coding) and the feature-compression standard (VCM, Video Coding for Machines), which opens a new paradigm of compressing the features a machine will analyze rather than the picture quality humans see (e.g., transmitting only the information needed to detect a person in CCTV). As next-generation standards, AOMedia's AV2 and JVET's experimental model ECM are under development, and the efficiency·license competition between the two camps is expected to determine the codec landscape of the late 2020s.

Differentiation by latency characteristics is also pronounced. In domains where ultra-low latency is essential, such as video conferencing·screen sharing (WebRTC) or cloud gaming·XR, latency-inducing tools such as B-frames are avoided, and latency·complexity·error resilience are taken as selection criteria over compression efficiency. For this reason a reversal appears in which, even with somewhat lower efficiency, a light profile or low-latency setting is adopted, and reference structures to guard against network loss (e.g., long-term reference, SVC layered coding) are used together. Conversely VOD can spend time performing 2-pass·shot-based optimization, so it pursues efficiency to the extreme. In short, "real-time or non-real-time" fundamentally divides the codec tool set and settings.

6. Considerations and Implications

The adoption·transition of a video codec must be judged strategically by weighing the following together.

  • Hardware-decoding dependency (application strategy): The efficiency gain of a newer-generation codec is realized only when the target terminal has a hardware decoder. The transition roadmap should be built in line with the hardware-support penetration curve of SoCs·TVs·browsers, and a lower-codec fallback for unsupported terminals must be run in parallel.
  • The triangular trade-off of efficiency-complexity-latency (trade-off): Raising compression efficiency increases encoding complexity and latency. It is reasonable to separate codecs·profiles by use case, prioritizing efficiency for VOD and low latency·low complexity for real-time broadcast·video conferencing·cloud gaming.
  • License·ecosystem risk (outlook): The precedent in which HEVC's fragmented patent pools delayed adoption shows that codec selection is not a technical metric but a matter of total cost of ownership (TCO) and legal risk. When choosing between royalty-free (AV1/AV2) and standard codecs (VVC), patent structure·predictability should be taken as key variables.
  • Investment shift toward encoding intelligence (related technology): Since the codec replacement cycle is as long as ten years, short-term efficiency improvement comes from per-title·shot-based·VMAF-based encoding optimization and ABR-ladder tuning. Apart from codec transition, investment in the encoding pipeline·quality metrics has a large immediate cost-saving effect.
  • Preparing for new paradigms (outlook): VCM (video coding for machines)·neural codecs shake the existing assumption centered on "the picture quality humans see." In CCTV·autonomous driving·industrial video where AI analysis becomes the center of consumption, compression judged by analysis accuracy rather than perceptual quality is required, so this trend should be pre-reflected in mid- to long-term architecture design.
  • Interoperability·standard conformance (application strategy): Because a codec standardizes only the bitstream syntax and leaves the encoder implementation free, locking into a specific encoder creates quality·compatibility risk. Standard conformance verification and end-to-end interoperability testing of container·DRM·HDR metadata must be performed before transition, and in multi-terminal environments a playback matrix by codec·resolution·HDR combination should be verified in advance.

References

In one line: A video codec is a lossy-compression technology that removes spatial·temporal·statistical·psycho-visual redundancy through hybrid block-based coding (prediction → transform·quantization → entropy → in-loop filter); from H.264 → HEVC → VVC/AV1 efficiency has roughly doubled every ten years, yet actual adoption is governed by hardware decoding·license·ecosystem.