2

I have a video which i want to cut and crop using opencv and ffmpeg.

diagram what i am trying to do

I want the output to be H265, so i am using a ffmpeg subprocess (writing frame bytes to stdin) as explained here. This is a minimum version of my code that leads to the error:

import os, shlex, subprocess, cv2, imutils

VIDEO_DIR = '../SoExample' # should contain a file 'in.mpg'
TIMESPAN = (3827, 3927)  # cut to this timespan (frame numbers)
CROP = dict(min_x=560, max_x=731, min_y=232, max_y=418)  # crop to this area

#  calculate output video size
size = (CROP['max_x']-CROP['min_x'], CROP['max_y']-CROP['min_y'])  # (w,h)
# ffmpeg throws an error when having odd dimensions that are not dividable by 2,
# so i just add a pixel to the size and stretch the original image by 1 pixel later.
size_rounded = (size[0]+1 if size[0] % 2 != 0 else size[0],
                size[1]+1 if size[1] % 2 != 0 else size[1])

# read input video
vid_path_in = os.path.join(VIDEO_DIR, 'in.mpg')
cap = cv2.VideoCapture(vid_path_in)
fps = int(cap.get(cv2.CAP_PROP_FPS))

# generate and run ffmpeg command
ffmpeg_cmd = (f'/usr/bin/ffmpeg -y -s {size_rounded[0]}x{size_rounded[1]} -pixel_format'
              + f' bgr24 -f rawvideo -r {fps} -re -i pipe: -vcodec libx265 -pix_fmt yuv420p'
              + f' -crf 24 -x265-params "ctu=64" "{os.path.join(VIDEO_DIR, "out.mp4")}"')
print("using cmd", ffmpeg_cmd)
process = subprocess.Popen(shlex.split(ffmpeg_cmd), stdin=subprocess.PIPE)

# seek to the beginning of the cutting timespan and loop through frames of input video
cap.set(cv2.CAP_PROP_POS_FRAMES, TIMESPAN[0])
frame_returned = True
while cap.isOpened() and frame_returned:
    frame_returned, frame = cap.read()
    frame_number = cap.get(cv2.CAP_PROP_POS_FRAMES) - 1

    # check if timespan end is not reached yet
    if frame_number < TIMESPAN[1]:

        # crop to relevant image area
        frame_cropped = frame[CROP['min_y']:CROP['max_y'],
                              CROP['min_x']:CROP['max_x']]

        # resize to even frame size if needed:
        if size != size_rounded:
            frame_cropped = imutils.resize(frame_cropped, width=size_rounded[0],
                                           height=size_rounded[1])

        # Show processed image using opencv: I see no errors here.
        cv2.imshow('Frame', frame_cropped)

        # Write cropped video frame to input stream of ffmpeg sub-process.
        process.stdin.write(frame_cropped.tobytes())
    else:
        break

    # Press Q on keyboard to exit earlier
    if cv2.waitKey(25) & 0xFF == ord('q'):
        break

process.stdin.close()  # Close and flush stdin
process.wait()         # Wait for sub-process to finish
process.terminate()    # Terminate the sub-process

print("Done!")

Unfortunately, my output looks like this:

gif of the sliding glitch

The output should not include this vertical sliding glitch. Does anyone know how to fix it?

My console output for aboves script shows:

using cmd /usr/bin/ffmpeg -y -s 172x186 -pixel_format bgr24 -f rawvideo -r 23 -i pipe: -vcodec libx265 -pix_fmt yuv420p -crf 24 -x265-params "ctu=64" "../SoExample/out.mp4"
ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  WARNING: library configuration mismatch
  avcodec     configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared --enable-version3 --disable-doc --disable-programs --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libtesseract --enable-libvo_amrwbenc
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, rawvideo, from 'pipe:':
  Duration: N/A, start: 0.000000, bitrate: 17659 kb/s
    Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24, 172x186, 17659 kb/s, 23 tbr, 23 tbn, 23 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> hevc (libx265))
x265 [info]: HEVC encoder version 2.6
x265 [info]: build info [Linux][GCC 7.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-2 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 2 / wpp(3 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 1 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : CRF-24.0 / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 rskip signhide tmvp strong-intra-smoothing
x265 [info]: tools: deblock sao
Output #0, mp4, to '../SoExample/out.mp4':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: hevc (libx265) (hev1 / 0x31766568), yuv420p, 172x186, q=2-31, 23 fps, 11776 tbn, 23 tbc
    Metadata:
      encoder         : Lavc57.107.100 libx265
[rawvideo @ 0x564ebd221aa0] Invalid buffer size, packet size 51600 < expected frame_size 95976    
Error while decoding stream #0:0: Invalid argument
frame=  100 fps= 30 q=-0.0 Lsize=      36kB time=00:00:04.21 bitrate=  69.1kbits/s speed=1.25x    
video:32kB audio:0kB subtitle:0kB other streams:0kB global headers:2kB muxing overhead: 12.141185%
x265 [info]: frame I:      1, Avg QP:22.44  kb/s: 179.77  
x265 [info]: frame P:     29, Avg QP:24.20  kb/s: 130.12  
x265 [info]: frame B:     70, Avg QP:29.99  kb/s: 27.82   
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x265 [info]: consecutive B-frames: 20.0% 3.3% 16.7% 43.3% 16.7% 

encoded 100 frames in 3.35s (29.83 fps), 59.01 kb/s, Avg QP:28.23
Done!

As you can see, there is an error: Invalid buffer size, packet size 51600 < expected frame_size 95976 Error while decoding stream #0:0: Invalid argument, do you think this could be the cause to the shown problem? I am not sure, as in the end, it tells me all 100 frames would have been encoded.

In case you want to reproduce this on the exact same video, you can find actions1.mpg in the UCF Aerial Action Dataset.

I would greatly appreciate any help as i am really stuck on this error.

GlabbichRulz
  • 948
  • 9
  • 28
  • maybe someone knows how this kind of video glitch is called? This could help me searching for further solutions in case nobody knows a direct answer. – GlabbichRulz Jul 07 '20 at 00:06
  • 1
    Your app is not sending the data fast enough, or with interruptions, so ffmpeg is receiving a truncated packet and that is causing loss of vertical sync. Add -re before -i but this may not help. – Gyan Jul 07 '20 at 07:37
  • @Gyan thank you for the explanation. I have tried the `-re` flag (edited code above and run again), but the output remains the same. I also looked into the `-vsync` flag and tried values 0-2, but it had no effect. – GlabbichRulz Jul 07 '20 at 09:07

1 Answers1

0

The error was fixed easier then expected.

imutils.resize() only tries to evaluate either the width or height argument, not both, and tries to preserve the original aspect ratio of the image, while cv2.resize() gives you a free choice (w/o any constraints) [Source Code, Explanation]

In my case, this lead to an unexpected resizing behaviour. I told imutils to resize my (171, 186) image to (172, 186), but what it did was resizing to (172, 187), as it decided to rather keep the aspect ration then the wanted height, so it discard my submitted height.

The simple fix is to replace

imutils.resize(frame_cropped, width=size_rounded[0], height=size_rounded[1])

by

cv2.resize(frame_cropped, (size_rounded[0], size_rounded[1]))

Fixing the resizing helped preventing the Buffer Size Error and therefore causing no vsync errors.

GlabbichRulz
  • 948
  • 9
  • 28