The Streaming Codec Landscape in 2021

 

By any measure, 2020 was a hectic year for video codecs or the compression technologies that drive streaming video. This year saw the launch of two standards-based codecs (with another due soon) from the Moving Pictures Experts Group (MPEG), the first hardware support for the Alliance for Open Media’s AV1 codec, and continued deployment of HEVC/H.265. In this article, I’ll provide a status update for streaming codecs, discuss the new MPEG codecs, identify factors that will control their adoption, and make some predictions regarding 2021 and beyond.

Before getting started, let’s clarify some terms. Each codec is defined by a specification — such as HEVC/H.265 and AV1. Multiple codec developers, however, create versions of that codec, usually with discrete names — such as x264, which is an implementation of AVC/H.264, and x265, which is an implementation of H.265/HEVC. Similarly, there are at least five current versions of the AV1 codec from vendors such as the Alliance for Open Media, Cisco, Intel/Netflix, and Visionular.

As you would expect, different implementations from different vendors have different performance characteristics. For example, Cisco is tuning its AV1 codec for Webex — its videoconferencing system — and claims that its implementation operates “within a similar CPU footprint to AVC/H.264.” As you’ll see below, my testing of four other AV1 codecs showed a CPU footprint of 15-30 times x264 for encoding video on demand. When discussing a codec in general, it’s appropriate to use the generic name. When discussing actual codec performance, it’s more appropriate to refer to the specific version itself.

In addition, AVC/H.264 and HEVC/H.265 have two names because each codec was standardized by both MPEG and the International Telecommunication Union (ITU). Versatile Video Coding, or VVC, is also H.266 for the same reason. In this article, I’m going to use the ITU designation (AVC, HEVC, VVC) because it’s more familiar to most of you. Just remember that AVC is the same as H.264, and you should be OK.

 

Where We Are

There are three primary markets for video consumption: browser, mobile, and OTT / smart TV. In all of these markets. AVC is by far the dominant codec, whereas HEVC is primarily deployed by producers who target living room viewers with premium content, often using high dynamic range (HDR). Both AVC and HEVC were launched by MPEG/ITU, and they’re royalty-bearing.

VP9 and AV1 are open-source codecs. VP9 was released by Google, and AV1 was released by the Alliance for Open Media. Although both organizations claim that their codecs are royalty-free, a patent pool administrator named Sisvel has launched patent pools for both codecs, putting this claim in doubt. In terms of usage, VP9 and AV1 are primarily used by the top-of-the-pyramid companies — such as Netflix, YouTube, Facebook, and Amazon — but by relatively few others.

   

In general, as bandwidth costs continue to drop, most producers below the Netflix/Amazon/YouTube level adopt new codecs only when needed to serve new markets, such as smart TVs. They don’t seem to adopt new codecs because of potential bandwidth savings. Why is this? Because deploying new codecs adds to encoding costs, R&D costs, and storage costs while making caching in delivery infrastructure less efficient. In addition — at least in the case of VP9 — it seems that when AV1 launched, many producers who’d been considering VP9 decided to instead go with AV1.

Let’s look at the basic performance characteristics for the pre-2020 codecs, as shown in Table 1. Note that the quality and performance data for AVC, HEVC, and AV1 were pulled from this article; whereas this article shows  libvpx-vp9 (Google’s VP9 codec in FFmpeg) slightly behind HEVC in quality but with roughly similar encoding times.

To explain the quality line, the 35% efficiency shown for HEVC means that the x265 codec in this study produced the same quality as x264 at about a 35% lower data rate. Similarly, the best performing AV1 codec (Aurora1 from Visionular) produced the same quality as x264 at about 50% of the data rate.

 

Table 1. Comparing performance and quality for pre-2020 codecs.

Codec	x264	x265	libvpx-vp9	AV1
Source	MPEG	MPEG	Google	Alliance for Open Media
Quality compared to x264 (VMAF)	N/A	~35%	~30%	~50%
Encoding requirements compared to x264	N/A	2–15x	2–15x	15–30x

 

Note that the range in encoding requirements vary so significantly for two reasons. With x264, it was because we tested using two presets — Slow and Veryslow. In the first referenced article, encoding with x265 using the Slow preset delivered our test files in 45 seconds, which is about twice as long as the x264 AVC codec. But encoding using the Veryslow preset took about 5 minutes, which is 15 times longer than x264 (quality results for the Veryslow preset are shown in Table 1). The reason the AV1 results vary so much was that we tested four AV1 codecs in that article. The fastest (again Aurora1 from Visionular) produced the test files in about 5 minutes, and the slowest in around 11.5 minutes. The quality shown is for Aurora1.

For perspective, encoding time typically translates directly into encoding cost, which is why most cloud encoders charge 2–4 times more to encode HEVC than AVC, and 30–40 times more for AV1 than for AVC. In a live environment, encoding time drives up machine costs to transcode the live encoding ladder. For example, in a recent consulting project, our test machine could produce six full x264 encoding ladders, but only one with x265 and libvpx-vp9. Meanwhile, when I attempted to produce the same ladder, the Alliance for Open Media’s AV1 codec — aomenc — could only push a frame or two per second, which brings into doubt the short-term viability of AV1 for live transcoding, or even WebRTC.

 

Where They Play

Beyond the production side, we choose codecs when they play on the target platforms that we care about (Table 2). Although it’s the least efficient in terms of quality, AVC plays in every target platform, so you can encode once and distribute everywhere. HEVC doesn’t play in Chrome and Firefox, which disqualifies it for use for general browser-based playback, but it’s supported by iOS and Android as well as nearly universally on smart TVs and the most recent crop of OTT devices.

 

Table 2. Codec availability by target platform.

Codec	AVC	HEVC	VP9	AV1
Browser support	Universal	Safari, Edge	Chrome, Edge, Firefox	Chrome, Firefox, Edge
Mobile support	iOS/Android	iOS/Android	iOS/Android	Android
OTT / Smart TV	Universal	Universal in smart TVs, recent OTT devices	Universal in smart TVs, recent OTT devices	In some OTT devices and some TVs starting in 2020

 

VP9 is supported by Chrome, Edge, and Firefox — with iOS support for apps only — but not by Safari. AV1 plays in the same browsers as VP9 and on the latest Android devices. The first smart TVs with AV1 support launched in 2020. As I write this article in January 2021, AV1 is supported by Roku but not Apple TV, and it’s not on the Amazon Fire Stick. It will definitely take a while before a critical mass of AV1-capable devices is available in the living room.

Synthesizing this data, I would expect more AV1 adoption for browser-based streaming, supplanting AVC and VP9, and for mobile playback via apps. But the living room will stay with AVC and HEVC for the next 12–24 months. HEVC won’t make significant inroads into browser-based streaming unless and until Google and Mozilla support it in their respective browsers.

For live producers, AVC and HEVC are much better positioned than VP9 and AV1. VP9 never really caught on for live production, so there are few hardware options for encoding or transcoding. One would expect AV1 to accumulate more hardware support for both, but at present, AV1 seems too complex for software-based transcoding.

 

What Happened in 2020?

This past year saw the launch of two MPEG codecs — VVC and Low Complexity Enhancement Video Coding (LCEVC) — and the continued development of Essential Video Coding (EVC). You can read more about VVC’s technical background here. I reviewed Fraunhofer’s VVC encoder and decoder, respectively called VVenC and VVdeC, for Streaming Media Magazine here. As Table 2 shows, VCVenC proved 39% more efficient than x265 in terms of quality, but it took 10x longer to encode, and it was 3.7x slower during decode.

Regarding royalties, the intellectual property that was contributed to VVC came from many parties, and VVC will be royalty-bearing, although we probably won’t know what the royalty policy will look like until mid-2021 or later.

 

Table 3. Benchmarking MPEG’s new codecs.

Codec	VVC	LCEVC	EVC
Finalized	July 2020	October 2020	Baseline Pending	Main Pending
Quality	39% x265	45% x264	31% AVC	27% HEVC
Encoding complexity	10x x265	-2.6x x264	1.4x AVC	4.5x HEVC
Decoding complexity	3.7x x265	1x x264	1.16x AVC	1.54x HEVC

 

LCEVC is a hybrid technology that uses an existing codec such as AVC or VP9 to produce a low-resolution base layer — for example, at 640×360 resolution for a 1080p file. Then, LCEVC adds an enhancement layer above the base layer to boost resolution to the full target and add quality. The base layer can play on any system that supports the base-layer codec, so an LCEVC file encoded with an AVC base layer should play at the base-layer resolution on any system that can play AVC, even without a LCEVC decoder. Add an LCEVC decoder to the system, and you get full resolution and quality.

 

 

V-Nova is the company that owns most of the intellectual property behind LCEVC, and I helped benchmark current performance in a report you can read here. Overall, an LCEVC file with a 640×360 base x264 layer and enhancement layer boosting resolution to 1920×1080 delivered the same quality as x264 at 1080p at a 45% lower bitrate. LCEVC encoding times were about 2.6x faster than x264, and decoding CPU requirements were about the same. V-Nova expects that this advantage will be preserved with other base-layer codecs so that LCEVC with HEVC as a base layer will be more efficient than HEVC only, but that will need to be verified on a codec-by-codec basis. LCEVC will be royalty-bearing, which I’ll discuss below.

Essential Video Coding has two profiles, and neither is finalized. The Baseline profile is built using older encoding techniques no longer protected by patents, and it’s supposed to be royalty-free. The Main profile is built from technologies from four companies, and it will bear a royalty. You can see the performance data for both profiles in Table 2.

 

When Will These New Codecs Become Relevant?

As I mentioned earlier, a codec becomes relevant when it’s available for playback on platforms that you care about. Several factors influence availability.

First is the technical side. If the codec needs hardware decoding available in the CPU/GPU to play smoothly or without too much battery drain, this is typically a two-year process from the date of codec launch — one year for CPUs, GPUs, and SoCs to incorporate the new codec, and another year for devices based on these chips. Both VVC and EVC seem to fall into this category, which means that the first consumer devices with VVC hardware may appear in mid-2022 at the earliest. Assuming that EVC is finalized in early 2021, devices with EVC support may appear, in the best-case scenario, in early 2023.

Because encoding and decoding LCEVC is so lightweight, LCEVC doesn’t need dedicated hardware for playback. This allows V-Nova to enable LCEVC playback in browsers, mobile devices, OTT devices, and smart TVs via software libraries or Javascript, requiring nothing from the manufacturer.

 

Known Royalty Policy

What could delay VVC and EVC even further is the release of the royalty policy associated with these technologies. Some chip vendors may refuse to start adding support for these codecs until they understand the business model and cost structure, which is a residue of the messy HEVC royalty policies involving three patent pools launched after many companies had already started deploying HEVC.

VVC IP owners are working toward finalizing a royalty structure sometime in 2021, which would only delay the two-year cycle by a few months. But until we know the proposed royalty numbers, we can’t assess VVC’s commercial viability. Regarding EVC, the companies that contributed intellectual property to EVC have pledged to announce their royalty policy within two years after the codec is finalized, which means as late as 2023.

LCEVC will also be royalty-bearing, but V-Nova plans to charge the companies that distribute the content, not the companies that supply encoding and decoding. In contrast, most other IP owners charge for encode/decode, which adds a royalty to every supporting phone, tablet, TV, or even browser. V-Nova’s reasoning is that by charging the content companies, it’s charging the companies that actually benefit from the reduced bandwidth costs. This schema should make it easier for content companies to expand the installed base of compatible LCEVC players because they could distribute a player with their app or browser code without paying an additional royalty.

 

Where Does It Play?

Beyond the royalty policy, whether the codec is available natively in a browser or mobile operating system can also dictate its use. Although Google supports AVC and HEVC in Android, it once threatened to remove AVC from Chrome, and it still doesn’t support HEVC in the world’s most popular browser, which is one of the key competitive advantages of AV1 over HEVC. Unless and until Chrome supports HEVC, it seems unlikely that Chrome will support VVC, EVC, or even LCEVC — and Google may not support them in Android.

Apple jumped on the HEVC bandwagon in June 2017 but joined the Alliance for Open Media (AOM) in 2018. It still doesn’t support either VP9 or AV1 in the Safari browser, and it’s impossible to predict whether it will embrace the three new MPEG codecs. In OTT / smart TV, VVC and EVC will first have to convince chip vendors to support their technologies, and then consumer-product vendors. Both groups include very prominent AOMedia members who may be reluctant to support new MPEG technologies. For example, chip vendors in AOMedia include Intel, AMD, and NVIDIA. Consumer-product vendors include Apple (AppleTV), Google (Chromecast, Android TV), Amazon (Fire), and Microsoft (Xbox), as well as Samsung (phones, tablets, smart TVs). Although HEVC has good adoption through these companies’ devices, most of those decisions were made before AV1 was available. How amendable these companies will be to new MPEG-based codecs remains to be seen.

 

Table 3. Benchmarking MPEG’s new codecs.

Codec	VVC	LCEVC	EVC
Finalized	July 2020	October 2020	Baseline Pending	Main Pending
Royalty status known	~ July 2021	December 2020	Unknown	Unknown
Chrome support	Unlikely	2020 via V-Nova software	Unlikely	Unlikely
Mobile support	Not assured, but the soonest in 2023	2020 via V-Nova software	Not assured, but 2023 (best case)	Not assured, but 2023 (best case)
OTT / Smart TV	Not assured, but the soonest in 2023	Some 2020 via V-Nova software; native support not assured	Not assured, but 2023 (best case)	Not assured, but 2023 (best case)

 

Looking at Table 4, the only new codec that could have an impact in 2021 is LCEVC. With VVC and EVC, there’s simply too much that’s unknown about royalties and technology adoption to gauge their potential, with the two-year hardware development cycle required for playback obviating any need to reach any conclusions in the short term.

 

Summing Up

There are three reasons to adopt a new codec: to compete in a new market such as smart TV with HDR, to harvest bandwidth savings, and to improve your viewer’s quality of experience. If you’re targeting the living room with premium HDR content, HEVC is the codec of choice, and will be for at least the next 1-2 years. If you’re chasing bandwidth savings or improved QoE, HEVC is the best choice for mobile and the living room, while AV1 is rapidly supplanting VP9 for browser-based playback.

Of the three new MPEG codecs, LCEVC is the only one that can have an impact in 2021-2022, and the best indicator of LCEVC’s viability will be deployments by top-of-the-pyramid publishers. The best short-term gauge of when VVC and EVC will become relevant is when their respective IP owners announce a royalty policy, and the completeness and cohesiveness of those policies. Ignore hyperbolic quality comparisons; if the economics don’t work, these codecs won’t gain broad-based adoption.