How to increase performance of OpenCV cv2.VideoCapture(0).read()

Question

I'm running this script on Kali linux with intel core i7-4510u:

import cv2
from datetime import datetime
vid_cam = cv2.VideoCapture(0)
vid_cam.set(cv2.CAP_PROP_FPS, 25)
vid_cam.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
vid_cam.set(cv2.CAP_PROP_FRAME_HEIGHT, 360)

lastDate = datetime.now().second
fcount = 0
while(vid_cam.isOpened()):
    if(datetime.now().second>lastDate):
        lastDate = datetime.now().second
        print("Fps: " + str(fcount))
        fcount = 0
    else:
        fcount += 1
    ret, image_frame = vid_cam.read()
    cv2.imshow('frame', image_frame)
    if cv2.waitKey(100) & 0xFF == ord('q'):
        break
vid_cam.release()
cv2.destroyAllWindows()

If I run it, it prints Fps: 4.
If I check Task Manager my cpu is at about 2%.
Where can the problem be?

Answer 1

Most often, people don't consider the data rate (bandwidth) required to get their data across the wire.

1920 x 1080 pixels at 30 FPS with 24 bits per pixel would require 1.5 Gbit/s. That won't fit through USB 2.

That is why many webcams implement compression. A common compression is MJPEG, which is JPEG as video.

You can tell OpenCV to tell the media API (V4L, dshow, MSMF, AVFoundation...) to request that from the camera. It won't on its own, I have learned.

# cap = cv.VideoCapture(...)
cap.set(cv.CAP_PROP_FOURCC, cv.VideoWriter_fourcc(*"MJPG"))

Less data per frame means the transfer goes faster, allowing more frames per time.

When mixing that with other properties (FRAME_WIDTH etc), order matters. I believe the FOURCC setting needs to come first, but if that doesn't help, try other orders.

When a webcam notices that there isn't enough free data rate on the USB controller, it may decide to reduce frame size or frame rate, or apply stronger compression (worse pictures). When you attach multiple cameras to the same USB hub, this will happen. This negotiation may cause opening to take several seconds, a minute even. If opening the camera is slow, maybe the USB controller already has some bandwidth reservations for other devices.

---

What does affect frame rate is when your other code in that loop simply takes a lot of time! Can't work around that, except to make that part of the code faster.

What will not generally matter is moving the reading into its own thread. In the OpenCV case (with imshow), a thread makes no sense at all. A thread only makes sense if there's anything else to be done with the time ordinarily spent waiting, besides processing the next frame (e.g. a GUI loop). Communication with consumers needs to be proper. Busy loops aren't the solution.

In the case of actual GUIs (not OpenCV's imshow), the proper thing to do is a thread, but it needs to update the widget as a frame comes in. There must not be any spinning loop that repeatedly reads a variable and assigns a widget's image or anything.

The camera will produce frames at its own pace (no matter what you do), and put them into a queue. Reading a frame costs fairly little (but not nothing).

You must read from the camera, or else the frames queue up. If you read slower than the camera's frame rate, you'll see increasing latency, i.e. movement in front of the camera takes seconds to show up on your screen.

If you try to read quicker, the read() call will just block until a frame is available. Then you'll "waste" time in that call. What would you do with that time? You don't have new data yet, so there's nothing to compute, until that frame arrives. Might as well wait, right?

Answer 2

One potential reason could because of I/O latency when reading frames. Since cv2.VideoCapture().read() is a blocking operation, the main program is stalled until the frame is read from the camera device and returned. A method to improve performance would be to spawn another thread to handle grabbing frames in parallel instead of relying on a single thread to grab frames in sequential order. We can improve performance by creating a new thread that only polls for new frames while the main thread handles processing the current frame. Here's a snippet for multithreading frames.

from threading import Thread
import cv2, time

class VideoStreamWidget(object):
    def __init__(self, src=0):
        self.capture = cv2.VideoCapture(src)
        # Start the thread to read frames from the video stream
        self.thread = Thread(target=self.update, args=())
        self.thread.daemon = True
        self.thread.start()

    def update(self):
        # Read the next frame from the stream in a different thread
        while True:
            if self.capture.isOpened():
                (self.status, self.frame) = self.capture.read()
            time.sleep(.01)

    def show_frame(self):
        # Display frames in main program
        cv2.imshow('frame', self.frame)
        key = cv2.waitKey(1)
        if key == ord('q'):
            self.capture.release()
            cv2.destroyAllWindows()
            exit(1)

if __name__ == '__main__':
    video_stream_widget = VideoStreamWidget()
    while True:
        try:
            video_stream_widget.show_frame()
        except AttributeError:
            pass