The Video Codec Landscape 2022

 

Over the last three years, many streaming producers went from working with a single codec, H.264, to no less than seven options: H.264, VP9, HEVC, AV1, VVC, LCEVC, and EVC. This article will update you on the status of each in 2022.  

I won’t go into detail about each codec in the interest of time and space. For a good background on the first four, check out last year’s The Streaming Codec Landscape in 2021. For more on the last three, Versatile Video Coding (VVC), Low Complexity Enhancement Video Coding (LCEVC), and Essential Video Coding (EVC), read Inside MPEG’s Ambitious Plan to Launch 3 Video Codecs in 2020. This article will liberally borrow performance data from Testing EVC, VVC, and LCEVC: How Do the Latest MPEG Codecs Stack Up, so you might glance at that article if you have any questions.  

I will make one preliminary point about EVC. Unlike all the other codecs with a single version, EVC has two: the Baseline and Main profiles. The Baseline profile is supposed to be royalty-free and designed to replace the H.264 codec, while the Main profile is royalty-bearing and targets replacing HEVC. 

Speaking of royalties, I won’t detail them in this article because they impact so few streaming publishers. Check out the following links to learn more about each codec’s royalty status: H.264, HEVC, VP9 (and here), AV1 (and here), VVC, LCEVC, EVC. 

   

Codec Quality

The essential function of a video codec is to compress video as efficiently as possible. Figure 1 shows the efficiency of each codec as compared to x265 from the Stack Up article. All the usual caveats apply; I tested several codec implementations for the article, so the results are for that codec implementation, and results will vary for other implementations. In addition, I didn’t test VP9 for the article, but I’ve found VP9 and HEVC to be relatively close in encoding quality in several analyses (as did Bitmovin here).

 

At a high level, VVC will be about 40%+ more efficient than HEVC, closely followed by the EVC Main profile and AV1. As an “Enhancement codec,” LCEVC quality will vary by base layer; with x265 as a base layer, LCEVC was about 22% more efficient than x265. The EVC Baseline profile proved significantly more efficient than x264, which is the clear laggard in the group — unsurprising for a codec that shipped in 2003. H.264 is wonderfully compatible over the broadest possible range of devices but costs you significantly in bandwidth with every stream you deliver. 

As a final caveat, even though AV1 is 38% more efficient than HEVC, that doesn’t mean you’ll save 38% of bandwidth costs if you deploy it. Check out Computing Breakeven on Codec Deployments here to learn why. 

 

Encoding Costs

One of the significant concerns with newer codecs is that the added complexity will cause encoding costs to skyrocket. Figure 2 shows the encoding times for the codecs tested in the Streaming Media article, which I added VP9 to afterward. Times are normalized with the x265 times at 100%. Encoding times translate directly to encoding costs for companies that run their own encoding farms and directly influence cloud encoding costs.

For those running their own encoders, AV1 should cost about the same as HEVC, a stunning improvement over the glacial times recorded about two years previously, while VVC should cost about two times HEVC. There was lots of Fear, Uncertainty, and Doubt (FUD) spread about the encoding time and cost of VVC, which seems unjustified at this point. 

The EVC Main profile was the laggard, but roughly 8x is pretty good for software that had only been available for a few weeks when I tested it. Notably, encoding LCEVC with x265 as a base layer costs 70% less than x265 while delivering 22% higher efficiency. Tastes great and has less filling!

 

While on the topic of encoding, let’s look at the timeline for live encoding and transcoding hardware. In general, it takes about a year after finalizing a specification for chipsets based on the spec to become available, followed a year later by actual encoders. Product options are limited for the next year or two, but encoders become commodities after five years or so. 

Figure 3 places each technology roughly within that adoption cycle, with H.264 and HEVC the only two codecs reaching commodity status. One or two hardware VP9 transcoders have been available for several years, but the VP9 live market never took off and likely won’t because most manufacturers are now focusing on AV1.

Note that there are no EVC chipsets announced. Like all things EVC — other than the excellent open-source encoder — the market for live EVC encode and transcode may never take off.

LCEVC can operate efficiently with existing hardware encoders and software encoders, so the typical hardware design cycle doesn’t impact it. Click here for a list of encoding options announced by V-Nova, LCEVC’s primarily technology contributor, at NAB in 2022.  

Both VVC and AV1 have one or two shipping encoders or transcoders in 2022 that prove the concept (and for which the developers will rightfully bang the PR drum loudly). There may be a handful of top-of-the-pyramid broadcasters considering live broadcasts or services based upon these codecs in 2022 and 2023, but the live streaming mass market won’t start for these codecs until much, much later. 

The big live streaming exception is AV1 for WebRTC, the browser-based real-time conferencing technology that H.264 has dominated with a smattering of VP9. Improbably for a technology that once encoded a frame every minute or so, several vendors, including Visionular and Cisco, claim to have real-time encoders for WebRTC shipping. You can expect to see a lot about that in 2022 and 2023. 

 

Playback Side

The main factor driving codec adoption is where it plays, not how easy it is to encode. The simplest playback market to enter is always browser-based playback on computers and mobile devices. Ask the two questions shown in Figure 4 to determine codec suitability for these devices.

The first question is whether the codec can play without hardware acceleration. If the answer is no, the codec won’t be usable until hardware support is widely available, which takes four to five years. H.264, HEVC, and VP9 already enjoy widespread hardware support on mobile devices and computers, though they can also play efficiently on devices without hardware support. 

AV1 and LCEVC aren’t widely supported but play efficiently in software, so they don’t need hardware support. This is why YouTube, Netflix, Vimeo, and other streaming producers have been distributing AV1 streams for several years, and several services for mobile phones and set-top boxes use LCEVC. Interestingly, after elbowing its way into becoming a “founding member” of AOMedia (27 months after AOMedia was actually formed), Apple still doesn’t support AV1 on any of its platforms in hardware or software.

With VVC, playback in software depends upon who you ask. In my tests of the Fraunhofer VVC player, playback was too slow for software-only, as were both EVC profiles. However, several contributors to the VVC technology, including Tencent and ByteDance, have impressive trials that show very efficient playback on mobile devices. One service in India claimed to be using VVC as early as June 2021. 

Still, even if VVC plays in software, it can’t pass the second test. This test is native playback support in the browser or operating system. VVC may ultimately play in Apple operating systems and browsers because Apple owns the only browser and operating systems that support HEVC natively and owns many VVC-related patents. However, it will likely never play in Chrome, Android, or Microsoft operating systems. 

Who so? Microsoft and Google, in particular, are staunch Alliance for Open Media (AOMedia) members and AV1 supporters. The “AOMedia lockout” that impacted HEVC will likely affect all subsequent MPEG-based codecs. As a practical matter, this means that any MPEG or similar codec will never be widely deployed for computer playback and will only deploy on mobile devices in apps, where decoders and the associated royalties are more controllable.  

 

Hardware Accelerated Playback – Mobile

Figure 5 shows the rough status of each codec respecting hardware playback support in mobile devices. Like in the encoder market, the adoption cycle roughly works as follows; once the specification is finalized, it takes a year to achieve chip support and another year for products based upon those chips to appear in the marketplace. Then it takes two to three years for the installed base of those products to grow into a market that’s relevant for general-purpose distributors. Of course, if you operate a dedicated service where you control both the encoder and decoder, typically through a dedicated set-top box, you can implement the codec once decoders are available. 

As shown in Figure 5, H.264 and H.265 are both pervasively supported, with Android widely supporting VP9. Apple added VP9 support in 2020, so that’s in year three of deployment — though it’s unclear whether this is hardware or software support. Either way, VP9 plays very efficiently on most reasonably current platforms. It, therefore, doesn’t need hardware support to achieve real-time playback or save battery life.

As previously mentioned, AV1 and LCEVC play efficiently in software, so they don’t require hardware support. AV1 is in year three of that adoption cycle (with a handful of phones with AV1 hardware decode) but burgeoning support in mobile chipsets. No mobile chipsets have been announced with VVC support, which pushes hardware-accelerated VVC support out into the 2026 range or beyond. As far as I can find, EVC has made little or no progress, so the timeline hasn’t even started.

 

Support in the Living Room

Figure 6 shows the same hardware adoption cycle for smart TVs, with EVC in the same “failure to launch” position. VVC enjoys both chipsets and one or two supporting TVs, pushing the addressable market to 2026 and beyond. AV1 is about a year ahead, and Netflix loudly started distributing AV1 streams to smart TVs in 2021, though these were SDR streams. There has been no announced support for HDR 10+ or DolbyVision for AV1, though both should be available once AV1-enabled smart TVs are widely deployed in 2024+.

Though Smart TVs widely support VP9, there’s no support for HDR10+ or DolbyVision, so the primary use is playing 2K, 4K, and 8K YouTube videos — not videos from premium content producers. Though LCEVC may play without hardware on some smart TVs, HDR support is similarly unproven. 

Few premium content producers currently distributing HEVC with Dolby Vision or HDR 10+ would switch back to SDR to save a tiny bit of bandwidth with AV1, LCEVC, or even VVC. For this reason, whenever you see a smart TV platform supports a particular codec, check the availability of dynamic-metadata-based HDR, which will be a prerequisite for most premium content.