Lossless compression of 10bit images (stored as 16bit pngs) with ffmpeg - HEVC preferably

Question

I am trying to encode 10 bit images losslessly in a video format, preferably using HEVC encoding. The images are stored as 16 bit png files (but only use 10 bit) and I have been working with ffmpeg to create and read back the video files.

My best attempt so far is based on https://stackoverflow.com/a/66180140/17261462 but as mentioned there, I get some pixel intensity differences which may be due to rounding when converting between 10 and 16 bit representation. I tried a few different means (bit shifting, left bit replication, floating point based scaling) but haven't yet figured out how to get a trully lossless reconstruction.

Below is a small piece of code to replicate my issue. I probably am doing something wrong there so feedback would be appreciated.

import subprocess
import numpy as np
import matplotlib.pyplot as plt
import tempfile
import imageio

# Create simple image
bitdepth = 10
hbd = int(bitdepth/2)
im0 = np.zeros((1<<hbd,1<<hbd),dtype=np.uint16)
im0[:] = np.arange(0,1<<bitdepth).reshape(im0.shape)
print('im0',np.min(im0),np.max(im0),im0.shape,im0.dtype)
# tile it to be at least 64 pix
im0 = np.tile(im0, (2, 2))
print('im0',np.min(im0),np.max(im0),im0.shape,im0.dtype)
im0ref = im0
# bitshift it or rescale intensities
#im0 = (im0<<6)
#im0 = (im0<<6) + (im0>>4)
im0 = np.uint16(np.round(im0 * np.float64((1<<16)-1)/np.float64((1<<10)-1)))
print('im0',np.min(im0),np.max(im0),im0.shape,im0.dtype)

# Save it as png
tmp0 = tempfile.NamedTemporaryFile(suffix='.png', delete=False)
print(f'Using tmp file: {tmp0.name}')
imageio.imwrite(tmp0.name,im0)

# Encode with ffmpeg
tmp1 = tempfile.NamedTemporaryFile(suffix='.mkv', delete=False)
# note that adding the following doesn't seem to impact the results 
#  + ' -bsf:v hevc_metadata=video_full_range_flag=1' \
mycmd = f'ffmpeg -y -i {tmp0.name}' \
  + ' -c:v libx265 -x265-params lossless=1' \
  + ' -pix_fmt gray10be' \
  + f' {tmp1.name}'
print(mycmd)
p = subprocess.run(mycmd.split(), capture_output=True)
print( 'stdout:', p.stdout.decode() )
print( 'stderr:', p.stderr.decode() )

tmp2 = tempfile.NamedTemporaryFile(suffix='.png', delete=False)
mycmd = f'ffmpeg -y -i {tmp1.name}' \
  + ' -pix_fmt gray16be' \
  + f' {tmp2.name}'
print(mycmd)
p = subprocess.run(mycmd.split(), capture_output=True)
print( 'stdout:', p.stdout.decode() )
print( 'stderr:', p.stderr.decode() )

# Read back with ffmpeg
im1 = imageio.imread(tmp2.name)
print('im1',np.min(im1),np.max(im1),im1.shape,im1.dtype)

# Bitshift or scale back
im1pre = im1
#im1 = (im1>>6)
im1 = np.uint16(np.round(im1 * np.float64((1<<10)-1)/np.float64((1<<16)-1)))

# check the result
plt.figure()
plt.imshow(im0ref)
plt.colorbar()

plt.figure()
plt.imshow(im1)
plt.colorbar()

plt.figure()
plt.imshow(np.int32(im1)-np.int32(im0ref))
plt.colorbar()

print('err: ',np.linalg.norm((np.float32(im1)-np.float32(im0ref)).ravel()))

plt.show()

EDIT: I have now also posted my question on the FFmpeg-user list: http://ffmpeg.org/pipermail/ffmpeg-user/2021-November/053761.html

Also for convenience, a simple script is provided below to generate the different variants of using the 16 bits with 10bit data:

import numpy as np
import imageio

# Create simple image with  gradient from
# 0 to (2^bitdepth - 1)
bitdepth = 10
unusedbitdepth = 16-bitdepth
hbd = int(bitdepth/2)
im0 = np.zeros((1<<hbd,1<<hbd),dtype=np.uint16)
im0[:] = np.arange(0,1<<bitdepth).reshape(im0.shape)

# Tile it to be at least 64 pix as ffmpeg encoder may only work
# with image of size 64 and up
im0 = np.tile(im0, (2, 2))
print('im0',np.min(im0),np.max(im0),im0.shape,im0.dtype)

# Save it
imageio.imwrite('gradient10bit-lsb.png',im0)

# Bitshift the values to use most significant bits
im1 = (im0<<unusedbitdepth)
print('im1',np.min(im1),np.max(im1),im1.shape,im1.dtype)
imageio.imwrite('gradient10bit-msb.png',im1)

# Scale the values use all 16 bits
im2 = np.uint16(np.round(im0 * np.float64((1<<16)-1)/np.float64((1<<bitdepth)-1)))
print('im2',np.min(im2),np.max(im2),im2.shape,im2.dtype)
imageio.imwrite('gradient10bit-scaledto16bits.png',im2)

# Left bit replication as a cost-effective approximation of scaling
# See http://www.libpng.org/pub/png/spec/1.1/PNG-Encoders.html
im3 = (im0<<unusedbitdepth) + (im0>>(bitdepth-unusedbitdepth))
print('im3',np.min(im3),np.max(im3),im3.shape,im3.dtype)
imageio.imwrite('gradient10bit-leftbitreplication.png',im3)

As well as raw ffmpeg / image magick commands.

Encoding:

ffmpeg -y -i gradient10bit-scaledto16bits.png -c:v libx265 -x265-params lossless=1 -pix_fmt gray10be gradient10bit-scaledto16bits.mkv

Decoding back to png:

ffmpeg -y -i gradient10bit-scaledto16bits.mkv -pix_fmt gray16be recons-gradient10bit-scaledto16bits.png

Comparison:

magick compare -verbose -metric mae gradient10bit-scaledto16bits.png recons-gradient10bit-scaledto16bits.png diff-scaledto16bits.png

Many thanks,

Tom

@Brad: The code snippet above creates a toy png image with intensities spanning the full 10bit dynamic range. I can't share one of my images at the moment but if you want something more photorealistic, you can easily create one from a 16 bit sample image found on the web, e.g. https://gist.github.com/tvercaut/f31dcc3180b80c7cda50ab23d0eceb06#file-synthetic10bit-py — Tom, Oct 28 '21 at 08:27

Mark Setchell · Answer 1 · 2021-11-04T14:15:28.373

If, as you say, you want to encode 10-bit images to video losslessly, you would surely do better to use a lossless format that is capable of storing such things - such as ffv1 - then you can store a full 16-bits without shifting/scaling or doing anything.

#!/bin/bash

# Generate 16-bit greyscale PNG
magick -size 1920x1080 xc:gray +noise random 1.png
magick 1.png -format "File: %f Unique colours: %k, Min: %[min], Max: %[max]\n" info:

# Encode to video
ffmpeg -v warning -y -i 1.png -c:v ffv1 -pix_fmt gray16le video.mkv

# Decode back to PNG
ffmpeg -v warning -y -i video.mkv 2.png
magick 2.png -format "File: %f Unique colours: %k, Min: %[min], Max: %[max]\n" info:

# Compare
magick compare -verbose -metric ae {1,2}.png null:

Output

File: 1.png Unique colours: 65536, Min: 0, Max: 65535
File: 2.png Unique colours: 65536, Min: 0, Max: 65535
1.png PNG 1920x1080 1920x1080+0+0 16-bit Gray 3.96256MiB 0.030u 0:00.029
2.png PNG 1920x1080 1920x1080+0+0 16-bit Gray 4161790B 0.020u 0:00.017
Image: 1.png
  Channel distortion: AE
    gray: 0
    all: 0
1.png=> PNG 1920x1080 16-bit Gray 3.96256MiB 0.760u 0:00.064

Thanks @Mark. I have now clarified in my question that HEVC encoding is preferred in my use case. The HEVC standard allows for 10 bit data and ffmpeg supports it. My question is really about being able to feed 10bit (grayscale) images to ffmpeg rather than finding an alternative codec which supports 16 bit data. — Tom, Nov 07 '21 at 16:32

Tom · Answer 2 · 2021-11-07T17:05:48.690

Thanks to Paul B Mahol on the ffmpeg-user mailing list, I have been able to solve this while using temporary rawvideo files. A solution without temporaries would nonethless be preferrable.

# convert png to rawvideo in 16 bits
ffmpeg -y -i gradient10bit-lsb.png -f rawvideo -pix_fmt gray16le gradient10bit-lsb.raw

# convert rawvideo to hevc-mkv in 10 bits by tricking the rawvideo demuxer
# into thinking the input is a 10 bit video
ffmpeg -y -f rawvideo -pixel_format gray10le -video_size 64x64 -i gradient10bit-lsb.raw -c:v libx265 -x265-params lossless=1 -pix_fmt gray10le gradient10bit-lsb.mkv

# delete tmp file
rm -f gradient10bit-lsb.raw

# convert hevc-mkv to rawvideo 10 bit
ffmpeg -y -i gradient10bit-lsb.mkv -f rawvideo -pix_fmt gray10le gradient10bit-lsb-postmkv.raw

# convert rawvideo back to png 16bits by tricking the rawvideo demuxer
# into thinking the input is 16 bits
ffmpeg -y -f rawvideo -pixel_format gray16le -video_size 64x64 -i gradient10bit-lsb-postmkv.raw -pix_fmt gray16be recons-gradient10bit-lsb.png

# delete tmp file
rm -f gradient10bit-lsb-postmkv.raw

# compare
magick compare -verbose -metric mae gradient10bit-lsb.png recons-gradient10bit-lsb.png diff-lsb.png

Lossless compression of 10bit images (stored as 16bit pngs) with ffmpeg - HEVC preferably

2 Answers2

Linked