HOW TO MEASURE VMAF

Our quality and performance results are all generated via the open-source tool ffmpeg. You can download this tool for free and it supports all of the latest GPUs with the current nightly builds—the latest official 5.1.2 release is from September 2022 and only has AV1 support for Intel’s Arc GPUs right now. ffmpeg supports VMAF, Netflix’s Video Multi-Method Assessment Fusion algorithm. Scores in VMAF range from 0 to 100, with 100 being excellent, 80 being good, 60 fair, 40 poor, 20 bad, and 0 representing a complete failure.

For our testing, we chose to target specific bitrates, since that’s important for streaming. We tried to improve the quality where possible, provided it didn’t kill the performance. AMD, for example, has three quality presets available—speed (the default), balanced, and quality.

We opted for the balanced mode, which was only slightly slower than speed but tended to improve the VMAF score by 5 to 10 percent. However, quality mode cut speed in half and only provided another one or two percent improvement, so we didn’t use that. We also specified a key frame interval of two seconds (-g 120), to try to level the playing field and make the resulting ‘streams’ as universally supported as possible.

Here are the specific commands we used for our encoding and VMAF tests, where [source] was a highquality 100Mbps H.264 video (40Mbps for 1080p), and [encoder] varies by GPU or CPU. For the CPU, we used libx264, libx265, and libsvtav1. AMD GPUs used h264_amf, hevc_amf, or av1_amf; Intel GPUs used h264_qsv, hevc_ qsv, or av1_qsv; and Nvidia GPUs used h264_nvenc, hevc_ nvenc, or av1_nvenc. [output] is the destination encoded file, and the other bracketed items are instructions for VMAF. ffmpeg -i [source] -c:v [encoder] -b:v [ bitrate] -g 120 -y [output] ffmpeg.exe -i [source] -i [output] -lavfi [0:v]setpts=PTSSTARTPTS[reference];[1:v] setpts=PTSSTARTPTS[distorted]; [distorted] [reference]libvmaf -f null