FFmpeg cant recognize 3 channels with each 32 bit

Question

I am writing the linearized depth buffer of a game to openEXR using FFmpeg. Unfortunately, FFmpeg does not adhere to the openEXR file specification fully (like allowing unsigned integer for one channel) so I am writing one float channel to openEXR, which is put into the green channel with this command -f rawvideo -pix_fmt grayf32be -s %WIDTH%x%HEIGHT% -r %FPS% -i - -vf %DEFVF% -preset ultrafast -tune zerolatency -qp 6 -compression zip1 -pix_fmt gbrpf32le %NAME%_depth_%d.exr.

The float range is from 0F to 1F and it is linear. I can confirm that the calculation and linearization is correct by testing 16 bit integer (per pixel component) PNG in Blender compositor. The 16 bit integer data is written like this short s = (short) (linearzieDepth(depth) * (Math.pow(2,16) - 1)) whereas for float the linearized value is directly written to OpenEXR without multiplying with a value.

However, when viewing the openEXR file it doesn't have the same "gradient" as the 16 bit png... when viewing them side by side, it appears as if the values near 0 are not linear, and they are not as dark as they should be like in the 16 bit png. (And yes, I set the image node to linear), and comparing it with 3d tracking data from the game I cant reproduce the depth and cant mask things using the depth buffer where as with the png I can.

How is it possible for a linear float range to turn out so different to a linear integer range in an image?

UPDATE:

I now write 3 channels to the ffmpeg with this code

float f2 = this.linearizeDepth(depth);

buffer.putFloat(f2);
buffer.putFloat(0);
buffer.putFloat(0);

the byte buffer is of the size width * height * 3 * 4 -> 3 channels with each 4 bytes. The command is now -f rawvideo -pix_fmt gbrpf32be -s %WIDTH%x%HEIGHT% -r %FPS% -i - -vf %DEFVF% -preset ultrafast -tune zerolatency -qp 6 -compression zip1 -pix_fmt gbrpf32le %NAME%_depth_%d.exr which should mean that the input (byte buffer) is expecting 32 bit floats with 3 channels.

FFmpeg is somehow splitting up channels or whatever... could be a bug, could be my fault?

Are you sure linearizeDepth is big endian? Are you sure the 3 color channels are written to the buffer as 3 planes one after the other sequentially (green plane width * height * 4 bytes, then blue plane width * height * 4 bytes then red plane width * height * 4 bytes)? What viewer are you using for displaying the exr image? — Rotem, Apr 04 '22 at 21:40
Yes, I was about to write that, I managed to fix it yesterday in the evening and hadn't had the time to write it here. I found out openEXR saves the R G B channels as separate "images" so to say... I found it through tests, but couldn't find it well explained in the file specs of openEXR — Chryfi, Apr 05 '22 at 08:16

Rotem · Accepted Answer · 2022-04-04T21:24:27.337

The issue is the color conversion from grayf32be to gbrpf32le.

Assuming source pixel range is [0, 1] we may add format conversion filter: -vf format=rgb48le before converting the pixel format to gbrpf32le.

It also looks like FFmpeg ignores the range arguments, the fix is adding scale filter: scale=in_range=full:out_range=full.

Updated command:

ffmpeg -y -f rawvideo -pix_fmt grayf32be -src_range 1 -s 192x108 -i in.raw -vf "scale=in_range=full:out_range=full,format=rgb48le" -vcodec exr -compression zip1 -pix_fmt gbrpf32le -dst_range 1 out.exr

Reproducible example:

Create 16 bits Tiff image (used as reference):

 ffmpeg -y -f lavfi -i testsrc=size=192x108:rate=1:duration=1 -pix_fmt gray16le in.tif

Convert the Tiff to float (big endian):

 ffmpeg -y -src_range 1 -i in.tif -pix_fmt grayf32be -dst_range 1 -f rawvideo in.raw

Convert from raw to OpenEXR format:

 ffmpeg -y -f rawvideo -pix_fmt grayf32be -src_range 1 -s 192x108 -i in.raw -vf "scale=in_range=full:out_range=full,format=rgb48le" -vcodec exr -compression zip1 -pix_fmt gbrpf32le -dst_range 1 out.exr

Python code for comparing the differences:

img1 = cv2.imread('in.tif', cv2.IMREAD_UNCHANGED)
img2 = cv2.imread('out.exr', cv2.IMREAD_UNCHANGED)

green_ch = img2[:, :, 1]  # Green channel

max_abs_diff = np.max(np.abs(green_ch*65535 - img1.astype(float)))

The maximum difference is 3 (out of 65535 levels).
We may have to play a bit with the filters arguments...

Since there are issues with FFmpeg color conversion and range conversion (so it seems), there is a change that you are not going to get the desired results until the issues are fixed.

Update:

Looks like it's working when the pixel format of the input is grayf32be (three color channels planar format).

Testing:

Create 16 bits Tiff image (used as reference):

 ffmpeg -y -f lavfi -i testsrc=size=192x108:rate=1:duration=1 -pix_fmt gray16le in.tif

Convert the Tiff to float (big endian):

 ffmpeg -y -src_range 1 -i in.tif -pix_fmt grayf32be -dst_range 1 -f rawvideo in.raw

Duplicate the "Grayscale plane" three times for getting 3 identical color planes (using "concat protocol" for avoiding any color conversion issues):
```
 ffmpeg -y -f rawvideo -pix_fmt grayf32be -s 192x108 -i "concat:in.raw|in.raw|in.raw" -f rawvideo in3.raw
```

Convert from 3 color channels raw to OpenEXR format:

 ffmpeg -y -f rawvideo -pix_fmt gbrpf32be -s 192x108 -i in3.raw -vcodec exr -compression zip1 -pix_fmt gbrpf32le out.exr

Python code for comparing the differences (compare 3 color channels):

img1 = cv2.imread('in.tif', cv2.IMREAD_UNCHANGED)
img2 = cv2.imread('out.exr', cv2.IMREAD_UNCHANGED)

blue_ch = img2[:, :, 0]  # Blue channel
green_ch = img2[:, :, 1]  # Green channel
red_ch = img2[:, :, 2]  # Red channel

max_red_abs_diff = np.max(np.abs(red_ch*65535 - img1.astype(float)))
max_green_abs_diff = np.max(np.abs(green_ch*65535 - img1.astype(float)))
max_blue_abs_diff = np.max(np.abs(blue_ch*65535 - img1.astype(float)))

The maximum difference is 0.001953125 (negligible).

but doesnt the format conversion to rgb48le scale down from 32 bit to 16 bit? Wont we then loose information? Once it has been cut down to 16 bit it doesn't matter if you scale up to 32 bit again, the other 16 bit are gone, yes? — Chryfi, Apr 04 '22 at 09:43
I can't fix FFmpeg... If 16 bits is not enough, try creating raw `gbrpf32le` input without pixel format conversion. Consider using OpenCV instead. — Rotem, Apr 04 '22 at 10:40
I'll try again passing 3 channels, all my previous tests resulted in a broken image. 16 bits is already cool, but the point of saving a depth buffer is having precision and openGL depth buffer is already 32 bit float. I'll also take a look at openCV, maybe it helps. FFmpeg has been annoying me ever since, especially since it doesn't adhere properly to the openEXR file specification. — Chryfi, Apr 04 '22 at 10:46
I updated the question with my new test results... I now pass 3 channels but ffmpeg decides to split them up even when declaring gbrpf32le as input pixel format... — Chryfi, Apr 04 '22 at 11:36
I have tried the following: **1.** `ffmpeg -y -f lavfi -i testsrc=size=192x108:rate=1:duration=1 -pix_fmt gray16le in.tif` **2.** `ffmpeg -y -src_range 1 -i in.tif -pix_fmt grayf32le -dst_range 1 -f rawvideo in.raw` **3.** `ffmpeg -y -f rawvideo -pix_fmt grayf32le -s 192x108 -i "concat:in.raw|in.raw|in.raw" -f rawvideo in3.raw` **4.** `ffmpeg -y -f rawvideo -pix_fmt gbrpf32le -s 192x108 -i in3.raw -vcodec exr -compression zip1 -pix_fmt gbrpf32le out.exr` There is a bug in FFmpeg - the float values are modified even without pixel format conversion (`in.raw|in.raw|in.raw` duplicates 3 times). — Rotem, Apr 04 '22 at 12:25
After re-checking, it looks like it's working when the pixel format of the input is `grayf32be`. I updated my answer. — Rotem, Apr 04 '22 at 21:26

FFmpeg cant recognize 3 channels with each 32 bit

1 Answers1

Update: