By using one the following command-lines, it is possible to convert a video stream to an RGB buffer:

ffmpeg -i video.mp4 -frames 1 -color_range pc -f rawvideo -pix_fmt rgb24 output.rgb24
ffmpeg -i video.mp4 -frames 1 -color_range pc -f rawvideo -pix_fmt gbrp output.gbrp

These RGB buffers can then be read, for example using Python and NumPy:

import numpy as np


def load_buffer_gbrp(path, width=1920, height=1080):
    """Load a gbrp 8-bit raw buffer from a file"""
    data = np.frombuffer(open(path, "rb").read(), dtype=np.uint8)
    data_gbrp = data.reshape((3, height, width))
    img_rgb = np.empty((height, width, 3), dtype=np.uint8)
    img_rgb[..., 0] = data_gbrp[2, ...]
    img_rgb[..., 1] = data_gbrp[0, ...]
    img_rgb[..., 2] = data_gbrp[1, ...]
    return img_rgb


def load_buffer_rgb24(path, width=1920, height=1080):
    """Load an rgb24 8-bit raw buffer from a file"""
    data = np.frombuffer(open(path, "rb").read(), dtype=np.uint8)
    img_rgb = data.reshape((height, width, 3))
    return img_rgb


buffer_rgb24 = load_buffer_rgb24("output.rgb24")
buffer_gbrp = load_buffer_gbrp("output.gbrp")

Theoretically, the two outputs should have the same RGB values (only the layout in memory should differ) ; in the real world, this is not the case:

import matplotlib.pyplot as plt

diff = buffer_rgb24.astype(float) - buffer_gbrp.astype(float)
fig, (ax1, ax2, ax3) = plt.subplots(ncols=3, constrained_layout=True, figsize=(12, 2.5))
ax1.imshow(buffer_rgb24)
ax1.set_title("rgb24")
ax2.imshow(buffer_gbrp)
ax2.set_title("gbrp")
im = ax3.imshow(diff[..., 1], vmin=-5, vmax= 5, cmap="seismic")
ax3.set_title("difference (green channel)")
plt.colorbar(im, ax=ax3)
plt.show()

The converted frame differs by more than what could be explained by chroma-subsampling or rounding errors (difference is around 2-3, rounding errors would be less than 1), and, what's worse, seems to have a uniform bias on the whole image.

Why is that so, and what ffmpeg parameters affect this behavior?

CodePudding user response：

Good analysis so far. Let me try to add some perspective from the swscale side, hope that helps further in explaining the differences you're seeing and what they technically originate from.

The differences you see are indeed caused by different rounding. These differences are not because rgb24/gbrp are fundamentally different (they are different layouts of the same fundamental data type), but because the implementations were written for different use cases at different times by different people.

yuv420p-to-rgb24 (and the other way around) are very, very old implementations that come from before swscale was part of FFmpeg. These implementations have MMX (!) optimizations and are optimized for optimal conversion on Pentium machines (!). This is mid-90s technology or so. The idea here was to convert JPEG and MPEG-1 to/from monitor-compatible output before YUV output was a thing. The MMX optimizations are actually pretty well-tuned for their time.

You can imagine that when speed is as important as it was here (and at that time, YUV-to-rgb24 conversion was slow). YUV-to-RGB is a simple

The ffmpeg source code uses the following functions internally to do the conversions in libswscale/output.c:

• yuv2rgb_full_1_c_template (and other variants) for rgb24 with full_chroma_int
• yuv2rgb_1_c_template (and other variants) for rgb24 without full_chroma_int
• yuv2gbrp_full_X_c (and other variants) for gbrp, independently of full_chroma_int

An important conclusion is that the full_chroma_int parameter seems to be ignored for gbrp format but not for rgb24 and is the main cause of the uniform bias.

Note that in non-rawvideo outputs, ffmpeg can select a supported pixel format depending on the selected format, and as such might get by default in either case without the user being aware of it.

An additional question is: are these the correct values? In other words, is it possible that both may be biased the same way? Taking the

There are remaining differences due to slightly different interpolation methods and rounding errors but no uniform bias, so the two implementations mostly agree up to that.

(Note: In this example the output.yuv file is in yuv444p, converted automatically from the native format of yuv420p by ffmpeg in the above command-line without going to the full RGB to YUV conversion. A more complete test would do all the previous conversions from a single raw YUV frame instead of a regular video to better isolate the differences.)