1

I noticed that OpenCV reads video frames almost 2x faster than FFMPEG.

Why is that? I thought all OpenCV does is call FFMPEG under the hood, possibly adding its own overhead.

Here's the code I use to obtain these results

import cv2
import time
import numpy as np

cap = cv2.VideoCapture("BigBuckBunny.mp4", apiPreference=cv2.CAP_FFMPEG)
frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

start = time.perf_counter()
while True:
    ret, frame = cap.read()
    if ret is False:
        break
    assert frame.shape == (720, 1280, 3)
    assert frame.dtype == np.uint8
end = time.perf_counter()

print(f"{frames/(end-start):.1f} frames per second")
# Output: 692.3 frames per second

cap.release()

For FFMPEG (using the python-ffmpeg library:

import ffmpeg
import numpy as np
import time

vid_info = ffmpeg.probe("BigBuckBunny.mp4")['streams'][1]
frames = int(vid_info['nb_frames'])

process1 = (
    ffmpeg
    .input("BigBuckBunny.mp4")
    .output('pipe:', format='rawvideo', pix_fmt='bgr24')
)
print(process1.compile())
# Output: ['ffmpeg', '-i', 'BigBuckBunny.mp4', '-f', 'rawvideo', '-pix_fmt', 'bgr24', 'pipe:']


process1 = process1.run_async(pipe_stdout=True)

start = time.perf_counter()
while True:
    in_bytes = process1.stdout.read(1280 * 720 * 3)
    if not in_bytes:
        break
    frame = np.frombuffer(in_bytes, np.uint8).reshape([720, 1280, 3])
end = time.perf_counter()
print(f"{frames/(end-start):.1f} frames per second")
# Output: 373.6 frames per second
process1.wait()

Here's information about the video (~10 minutes length)

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'BigBuckBunny.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isomavc1mp42
    creation_time   : 2010-01-10T08:29:06.000000Z
  Duration: 00:09:56.47, start: 0.000000, bitrate: 2119 kb/s
  Stream #0:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      creation_time   : 2010-01-10T08:29:06.000000Z
      handler_name    : (C) 2007 Google Inc. v08.13.2007.
      vendor_id       : [0][0][0][0]
  Stream #0:1(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 1991 kb/s, 24 fps, 24 tbr, 24k tbn, 48 tbc (default)
    Metadata:
      creation_time   : 2010-01-10T08:29:06.000000Z
      handler_name    : (C) 2007 Google Inc. v08.13.2007.
      vendor_id       : [0][0][0][0]

And FFMPEG and OpenCV versions:

ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers

opencv-python-headless        4.6.0.66
tadejsv
  • 2,085
  • 1
  • 18
  • 20
  • What sort of machine (CPU etc) is this running on please? – Mark Setchell Sep 17 '22 at 08:16
  • Lenovo Thinkpad X1 Carbon 6th gen, Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 16GB RAM, Ubuntu 22.04 – tadejsv Sep 17 '22 at 08:18
  • Did you check you get exactly the same number of frames and same data by both methods? I have not used the `run_async` stuff and wonder if it is in *"binary"* mode, or maybe translating linefeeds and carriage returns unless you use something like `sys.stdin.buffer.read` Example here https://stackoverflow.com/a/32282458/2836621 – Mark Setchell Sep 17 '22 at 08:35
  • Yes, they both give the same number of frames. I checked frames at some arbitrary positions (1200, 5000, 14000), and they were the same for OpenCV and FFPMEG. – tadejsv Sep 17 '22 at 08:41
  • 1
    I wonder if OpenCV is reading in a YUV format which has fewer bits/pixel (because of colour subsampling) and converting up to BGR24 using its optimised SIMD instructions. – Mark Setchell Sep 17 '22 at 08:49
  • Would be great if there was a way to test this :) – tadejsv Sep 17 '22 at 08:52
  • Btw @MarkSetchell, can you try replicating the results? You can find the link to the video here: https://gist.github.com/jsturgis/3b19447b304616f18657 – tadejsv Sep 17 '22 at 08:53
  • Can you try to deactivate hw_acceleration stuff in VideoCapture? https://docs.opencv.org/4.x/db/dc4/group__videoio__hwaccel.html if this leads to the same slow decoding as ffmpeg you can try to find the same acceleration in ffmpeg and activate it there. – Micka Sep 17 '22 at 09:26
  • @Micka, I tried doing that: `cap = cv2.VideoCapture("BigBuckBunny.mp4", apiPreference=cv2.CAP_FFMPEG, params=[cv2.CAP_PROP_HW_ACCELERATION, cv2.VIDEO_ACCELERATION_NONE])` - but there was no difference. Did I disable acceleration properly? – tadejsv Sep 17 '22 at 09:35
  • No idea how exactly it is done (especially in python), sorry. Docs say for ffmpeg the parameter is mapped to some ffmpeg enumerator values, but I dont even understand how to interpret that... – Micka Sep 17 '22 at 09:44
  • 1
    Holly molly @MarkSetchell you are right (or at least on to something)! If I change `pix_fmt='bgr24'` to `pix_fmt='yuv420p'` (and remove re-shaping of output for ffmpeg) the speed of ffmpeg is now slightly above OpenCV. If I have time I will try to see if I can add this conversion manually. Otherwise, feel free to write up an answer, I can mark it as correct :) – tadejsv Sep 17 '22 at 11:18

2 Answers2

1

It may be due to the fact how the libraries are implemented and the overhead of pipes.

  1. cv2

cv2 implemented in c++ and uses libavcodec library (part of ffmpeg) for decoding frames:

https://github.com/opencv/opencv/blob/97c6ec6d49cb78321eafe6fa220ff80ebdc5e2f4/modules/videoio/src/cap_ffmpeg_impl.hpp#L1299

cv2 does not use ffmpeg as binary executable and uses the part of ffmpeg specifically designed for reading/decoding frames.

  1. ffmpeg-python is a wrapper around ffmpeg executable (which in turn uses libavcodec under the hood too) that calls ffmpeg with subprocess and returns results through the pipes.
u1234x1234
  • 2,062
  • 1
  • 1
  • 8
  • What you say is true, but I find it strange that the difference would be this large. Not shown in the code above, but I also tried a case with `ffmpeg` batching the frames (using `vframes` output option), but it did not result in any increase of speed for ffmpeg – tadejsv Sep 17 '22 at 08:48
1

Good stuff. Interestingly, under Windows 10 & FFmpeg 5.0.1, my results are the reverse of yours (but much closer):

  • OpenCV: 491.9 frames per second
  • FFmpeg-Python: 519.4 frames per second

And out of curiosity, I tried my ffmpegio package to see how much overhead it incurs, it came out: 507.5 frames per second (code below) Both FFmpeg-Python and FFmpegIO scoring in the same ballpark indicates these numbers aren't fluke.

So, it's likely Win/Linux or 4.4/5.0 differences or both. I retested FFmpeg-Python with FFmpeg 4.4.1 and got: 428.1 frames per second (log reporting speed=17.9x down from x21.7. I'd say FFmpeg devs optimized the color conversion codebase in v5 releases to match or surpass OpenCV. (Or the tested OpenCV version uses the libav libraries associated with FFmpeg v5)

test_ffmpegio.py

import ffmpegio
import time

frames = ffmpegio.probe.video_streams_basic("BigBuckBunny.mp4")[0]['nb_frames']

print(frames)

with ffmpegio.open("BigBuckBunny.mp4",'rv') as f:
    start = time.perf_counter()
    for frame in f:
        pass # frame is 1x1280x720x3 rgb np array
    end = time.perf_counter()

print(f"{frames/(end-start):.1f} frames per second")
kesh
  • 4,515
  • 2
  • 12
  • 20