Video Codecs and Encoding: Everything You Should Know (Update)

 

Without video codecs, “Netflix and chill” wouldn’t have ever been coined. These two-part compression tools allow distributors to condense a video file for delivery across the internet via a process called video encoding. Codecs are the reason we can so easily conduct business meetings over Zoom and binge TV shows on our phones — even with limited bandwidth.

Thanks to codecs, Netflix manages to stream more than 450,000 hours of content every minute. And in order to get those streams to the various devices that end-users are tuning in from, Netflix must use both new and tried-and-true codecs.

What does video encoding involve and how do video codecs work? We’ll dig deeper below and cover our list of the best video codecs for streaming.

 

What Is Video Encoding?

Video encoding refers to the process of converting raw video into a digital format that’s compatible with many devices. When it comes to streaming, videos are often compressed from gigabytes of data down to megabytes of data. Video encoding is essential to live streaming, helping to ensure quick delivery and playback.

Encoding can occur within a browser or mobile app, on an IP camera, using software, or via a stand-alone appliance. Popular software options include Vmix, Wirecast, and the free-to-use OBS Studio.

In order to compress the raw video into a more manageable size, encoders use video and audio codecs, which apply algorithms to shrink the bulky video for delivery. To put it more simply: encoding describes the process of compression, whereas codecs describe the means for doing so.

 

What Is a Codec?

Content distributors use a video compression technology called a codec to shrink videos into a streamable size. Codecs allow us to tightly compress bulky streams down for delivery and storage.

Streaming requires the use of both audio and video codecs. H.264, also known as AVC (Advanced Video Coding), is the most common video codec; AAC (Advanced Audio Coding) is the most common audio codec.

But wait — why does H.264 also go by AVC? And how are we to sort through so many confusing acronyms? Streaming codec expert Jan Ozer explains it:

“H.264/AVC and H.265/HEVC have two names because each codec was standardized by both MPEG and the International Telecommunication Union (ITU). Versatile Video Coding, or VVC, is also H.266 for the same reason.”

For the sake of clarity, I’ll be including both names at first mention of each codec within the subsections of this article.

 

How Do Codecs Work?

Literally ‘coder-decoder’ or ‘compressor-decompressor,’ codecs apply algorithms to the video and create a miniature version of it. For that reason, these video compression standards are two-trick ponies: They facilitate both the encoding/compression and the decoding/decompressing aspect of video streaming.

 

Lossy vs. Lossless Video Encoding

Lossless compression involves compacting data without discarding any information. When you cram a bunch of .jpegs into a .zip file without compromising the data integrity, for instance, you’re employing lossless compression.

Lossy compression, on the other hand, gets rid of data. And when you use the .jpeg codec, you’re doing just that. Lossy codecs discard unnecessary data and reduce quality where possible. For that reason, a raw image converted to .jpeg will always be lower quality, but easier to store and share.

When it comes to streaming, lossy compression is the name of the game. Video codecs employ it to shrink down content for storage and transmission, and later decompressed the video for viewing. While the final video should appear the same, it will be missing any data that was deemed undetectable by the human eye.

Lossy compression is a lot like Wonkavision in Charlie and the Chocolate Factory. It makes a large collection of data smaller for transport to your screen:

 

 

What Is a Video Container Format?

Once compressed, the components of a stream are packaged into a wrapper or file format. These files contain the audio codec, video codec, closed captioning, and any associated metadata. Common containers include .mp4, .mov, .ts, and .wmv.

Containers can often input multiple types of codecs. That said, not all playback platforms accept all containers and codecs. That’s why multi-format encoding is crucial when streaming to a wide range of devices.

For example: a .mov file and a .wmv file might have the same data and codecs inside. But the .mov file would be used for playback on Macbook’s QuickTime player, while the .wmv file would be used for playback on a PC’s Windows Media Player.

 

Video Codecs vs. Containers: What’s the Difference?

A codec acts upon the video, both at the source to compress it and before playback to decompress it. This is done through lossy compression, during which any unnecessary data is discarded.

On the other hand, a video container format stores the video codec, audio codec, and metadata such as subtitles or preview images. The container holds all the components together and determines which programs can accept the stream.

 

A Note on Encoding Profiles

Each of the codecs described below is more accurately thought of as an umbrella for closely related encoding profiles. Newer technologies, like Essential Video Coding (EVC), only have two profiles. Whereas older technologies, like H.264, have enough profiles to fill this entire table.

   

Our discussion of the best video codecs will look at each codec in a general sense. But it’s worth noting that there are some differences among encoding software and hardware due to this variation.

 

A Brief History of Video Codecs

Data and video compression go back decades, but the International Telecommunication Union (ITU) developed the first commercially successful video codec, H.261, in 1988. The organization designed it for use in video conferencing and the codec provided a resolution of 352×288 pixels at a frame rate of 30 frames per second. The ITU also developed the first audio codec, G.711, in the 1970s, designed for use in telephony networks and had a bit rate of 64 kbps.  

In the 1990s, the Motion Picture Experts Group (MPEG) developed a series of video coding standards that became the most widely used codecs in the world. The first MPEG standard, MPEG-1, was developed in 1993 and was created for use in CD (VCD) players. It had a resolution of 352×240 pixels at a frame rate of 30 frames per second. Later, the organization developed MPEG-2 for use in DVD players with a similar frame rate but higher resolution of 720×480 pixels. The most widely used video codec since 2016 is H.264/MPEG-4 AVC, which is used in a variety of applications that record or transmit video. 

 

Best Video Codecs for Streaming

Delivering video over the internet to a variety of devices starts with encoding across a variety of codecs. Next-generation codecs improve encoding efficiency and quality, while legacy codecs enable playback on outdated machines.

Take it from the largest distributor of video streaming around: Netflix.

“Netflix says it utilizes a deep toolbox of codecs, which can be called upon to stream compatible formats to display devices. Although Netflix continually adds new and improved codecs, it has never abandoned one — it continues to support the VC1 codec it started with in the first Netflix streaming device, a 10-year-old LG Blu-ray player.”

Our video codec list below includes both the old and the new. While industry leaders continue to refine and develop the latest compression tools, they also employ older codecs like H.264/AVC for delivery to legacy devices. To put it another way, we live in a multi-codec world, with the landscape trending toward fragmentation rather than consolidation.

   

Let’s look closer at the top video codecs in 2022.

1. H.264/AVC
2. H.265/HEVC
3. AV1
4. VP9
5. H.266/VVC

 

H.264/AVC

The majority of encoding output today takes the form of H.264 files, also referred to as AVC (Advanced Video Coding). This widely supported codec was developed by the International Telecommunications Union and the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Moving Picture Experts Group — wow, what a mouthful.

H.264 also has significant penetration into markets outside of streaming, such as Blu-ray disks and cable broadcasting. It is often incorporated with the AAC audio codec and can be packaged into .mp4, .mov, .F4v, .3GP, and .ts containers.

H.264 plays on virtually any device, delivers quality video streams, and comes with the least concerns surrounding royalties. That’s not to say that it isn’t royalty-bearing, only that content publishers know what to expect in terms of royalties — which isn’t always the case. Because of its widespread device support, H.264 remains the most used option. That said, in Bitmovin’s 2021 Video Developer Report, it fell 8 percent in usage year over year (from 91 percent in 2020 to 83 percent in 2021).

Which video codecs are you using in production?

   

Although H.264 works great across all the primary markets for video consumption (browser, mobile, Smart TVs), it isn’t suitable for 4K video or high dynamic range (HDR) content.

 

Table 1: Video Codec Comparison: Suitability

Codec Suitability	Live Origination	Live Transcode	Low Latency	4K	HDR
H.264	Excellent	Excellent	Excellent	Poor	Poor
VP9	Poor	Poor	WebRTC	Excellent	Poor
HEVC	Good	Good	Nascent	Excellent	Excellent
AV1	Nascent	Nascent	WebRTC	Excellent	Nascent

 

Rather, as the speediest codec detailed in this blog, H.264 is better applied to low-latency streaming than 8K video delivery. Streaming Media’s Jan Ozer explains:

“H.264 excels in live origination and transcode because of encoding speed; all other codecs need hardware support for both functions. Understandably, H.264 has received the lion’s share of attention in low-latency applications, whether HTTP-based or WebRTC.”

H.264 is also the oldest video encoding format on this list. How old, you ask? H.264 shipped in 2003 — which means it’s almost reach drinking age in the U.S.

Many predicted it would be obsolete by now. But when you factor in its processing power and cost, H.264 maintains its allure. Plus, the number of devices that can encode and decode H.264 can’t be beat — including IP cameras, set-top boxes, mobile devices, and low-power devices.

 

TL;DR

H.264 is an efficient and widely adopted video compression technology used to contribute, distribute, and deliver streams — unparalleled in terms of device reach. It lends especially well to low-latency workflows.

   

H.265/HEVC

The ISO/IEV Moving Picture Experts Group developed H.265 as the successor to H.264. Also called HEVC (High Efficiency Video Coding), this codec aims to improve compression efficiency and support 8K resolution. It generates smaller files than H.264, thus decreasing the bandwidth required to view these streams. This makes it an ideal codec for high-resolution streaming.

That said, only about 10 percent of encoded files take the form of H.265. Uncertainties about royalties have stifled adoption. Specifically, content distributors are frustrated by the lacking transparency into what they’ll have to pay when using this codec.

The patent and royalty drama surrounding H.265 directly resulted in the development of the AV1 codec (which we’ll cover next) by the Alliance for Open Media, as well as its lacking compatibility for browser-based playback. Why? Industry leaders like Google, Microsoft, and Mozilla had no interest in adding support for the costly codec to Chrome, Edge, and Firefox. As a result, only ~18.08% of browsers accepted H.265-encoded video up until recently.

One place where H.265 remains a better option than VP9 or H.264 is 4K and HDR video delivery to living room devices, as it’s almost universally supported on Smart TVs.

 

Table 3: Video Codec Comparison: Royalty Status

Codec Royalty Status	Encoder	Decoder	Paid Content	Free Internet Content
H.264	Yes	Yes	Yes	No
VP9	No	Consumer device	No	No
HEVC	Yes	Yes	Some	Unclear
AV1	No	Consumer Device	No	No

 

And then all of the sudden, Google added H.265 support in Chrome, making it a viable option for browser-based video consumption. Because the codec is also universally supported on Smart TVs, we expect a major uptick in use.

With this change, 84% of browsers should be able to play video encoded with H.264 (assuming that the devices accessing those browsers have underlying support).

 

TL;DR

If you’re delivering premium OTT content to living room devices, H.265 is your best bet. And with Google’s added support for HEVC in Chrome, it’s poised to become a common technology for browser-based video streaming as well.

   

AV1

Frustrated about the royalties associated with H.265, Amazon, Netflix, Google, Microsoft, Cisco, and Mozilla formed the Alliance for Open Media. The goal? Create an open-source, royalty-free alternative called AV1.

This initiative to democratize high-quality video delivery and playback has been a long time coming. But in 2022, adoption is finally coming to fruition.

   

According to Johnathan Rosenberg, CTO of the Collaboration Technology Group at Cisco, “The creation of an advanced, royalty-free video codec is paramount to the ongoing success of collaboration products and services. This is why Cisco joined AOMedia as a founding member, and why Cisco has invested in making AV1 both efficient and accessible to the internet community.”

AV1 touts itself as being 30 percent more efficient than H.265. But it has taken some time for AV1 hardware decoding capabilities to be integrated on a mass scale. A lack of living room devices (a.k.a. Smart TVs) capable of decoding AV1 streams has long hampered adoption. That said, Fire TV sticks have added support, and other hardware manufacturers are following Amazons lead. Leaders at Netflix, YouTube, and Amazon have also made the move to AV1. Even more exciting, AV1 is finding its way into real-time encoding with WebRTC. Cisco, Google, and even Wowza’s Real-Time Streaming at Scale feature can encode AV1 — allowing the technology to make an impact in both the VOD and live space.

 

TL; DR

AV1 is the latest and greatest open-source video encoding technology. Adoption is finally coming to fruition for both OTT and real-time applications.

   

VP9

Google developed VP9 as a royalty-free, open-source alternative to H.265. The Google-owned YouTube platform and Chrome browser support VP9, as well every Android phone, Mozilla’s Firefox, Apple’s Safari, and all new iOS devices. This codec also makes an appearance in many WebRTC workflows, with more than 90% of Chrome-encoded WebRTC video using VP9 or its predecessor VP8.

VP9 was released in 2013, which puts it in the middle of the pack as far as age goes. Even so, it’s a better option than most for several reasons. For one, VP9 performs about the same as H.265/HEVC. This makes it well suited for 4K video, especially when publishing to YouTube.

Beyond that, VP9 ranks second only to H.264/AVC in terms of compatibility across browsers and devices. Samsung, Sony, LG, Roku, and many other household names support it. Plus, Google’s implementation of the codec in YouTube and Netflix’s use of it for some content will continue to drive this trend.

 

Table 2: Video Codec Comparison: Compatibility

Codec Compatibility	Browser	Mobile	Smart TV/OTT
H.264	Virtually all	All	All
VP9	Virtually all	Android, iOS	Most
HEVC	Very little	Android, iOS	All
AV1	Edge, Firefox, Chrome, Opera	Android	Nascent

 

Think of VP9 as AV0, or an earlier version of AV1. Both are open-source, and both claim to be royalty-free (although there is some doubt surrounding this). For this year’s list of the best video codecs, we’ve moved VP9 down a few spots because it’s no longer expected to take off given a new focus on AV1 by hardware manufacturers.

 

TL; DR

VP9 is a more advanced and higher-quality compression technology than H.264/AVC that enjoys greater compatibility than many of its alternatives and works well for 4K streaming. That said, the prioritization of AV1 support has bumped VP9 down in 2022.

   

H.266/VVC

As the newest kid on the video compression block, the H.266/VVC (Versatile Video Coding) specification was only just finalized in 2020. While intended to usurp H.265 and H.264, it has the same royalty issues as its predecessors.

Back in 2008, Beamr’s chief technology officer Dror Gill explained, “It’s okay to pay royalties as long as you know how much you need to pay and when. With H.264, it was very clear how much you need to pay, there was one body collecting all the royalties, and this became the world’s most prominent video codec. The same can happen with VVC, if they get their act together before releasing the standard.”

And yet, the H.266/VVC royalty rate remains a mystery at this time. Unforeseen challenges on the licensing front are up in the air, and we’re also waiting to see how adoption of the codec pans out. 

To put it in a nutshell à la codec expert Jan Ozer:

“Overall, the individual H.266/VVC codec developers have made great strides in delivering the promised bandwidth savings, though final performance won’t be known until the royalty policies are set and we know which tools are in which profiles. Beyond that, given the diverse range of other factors in play, it’s impossible to know at this time whether VVC will ever reach critical mass.”

 

TL; DR

Still a nascent technology, H.266/VVC won’t even be a consideration for most content publishers until 2022 at the earliest, with too many unknowns surrounding the codec to make any predictions.

   

Encoding Best Practices

Encoding best practices go way beyond what codec you select. You’ll also want to consider frame rate, keyframe interval, and bitrate.

Luckily, a live stream can always be transcoded into another format once it reaches the server. This can be done using streaming software and your own servers, or in the cloud for professionally managed delivery.

 

Encoding vs. Transcoding

So what exactly is transcoding? Transcoding involves taking an encoded file and decoding it to alter it in some way. This can take the form of re-encoding the data into a more common codec, transizing the video into a lower resolution, transrating the file into a different bitrate, or transmuxing it into a more scalable protocol.

Once the process is complete, the media server recompresses the manipulated file for delivery. 

Transcoding gives you the freedom to change an H.264-encoded video into a VP9- or H.265-encoded video. That way, your content would be optimized for 4K streaming to end users — while also benefiting from speedy video contribution.   

Think of encoding and transcoding as the steps involved when packing for a vacation. Initially, you compress (encode) your clothing into a bag for easy transportation to wherever you’re headed. Once you arrive at your destination, you unpack, discard items that you no longer need, add trinkets gathered during your travels, and then repack your bag for the next step of your journey. This is transcoding.

   

Multi-Codec Video Delivery

There you have it. Video codecs are what allow us to take the boundless world, capture a slice of it through the lens of our camera, and compress it down for delivery over the internet.

Because proprietary codecs and video containers exist, it’s essential to deliver multiple different versions of your live streams to viewers.

Luckily, we offer the Wowza Video platform for multi-codec video delivery across the globe. That way, you can convert streams as needed by transcoding the data into new codecs or transmuxing into different video encoding formats.