OpenCV real time streaming video capture is slow. How to drop frames or get synced with real time?

Question

Goal and problem

I'd like to set up an opencv system to process either HLS streams or RMTP streams, however, I am running into a strange issue regarding a reduced frame-rate and an accumulating lag. It's as if the video gets further and further behind from where it is supposed to be in the stream.

I'm looking for a way to keep up to date with a live source even if it means dropping frames.

Current approach

import cv2

cap = cv2.VideoCapture()
cap.open('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8')

while (True):
    _, frame = cap.read()
    cv2.imshow("camCapture", frame)
    cv2.waitKey(1)

I've validated the quality of the stream on VLC and it seems to work fine there.

cv2 speed

.

realistic/expected speed

Questions:

What am I doing wrong here?
Why is it so slow?
How do I sync it to real-time speeds?

On my machine, I get speeded up video for ~2sec with your code. Then a stop for ~2 sec. repeat. I tried to force the video stream to show the newest frame with cap.set(cv2.CAP_PROP_POS_AVI_RATIO, 1). It seems to work, but I get one frame every seconds. The FPS from print(cap.get(cv2.CAP_PROP_FPS)) is 180,000.0, which is way too high. But if you could get an accurate FPS, maybe you could do some jiggery pokery with multiple cap.grab() to advance to where you should be in the stream and then a single cap.retrieve to get the frame. — bfris, Oct 08 '19 at 21:23
I see that you have the minimum code to video acquisition in python. So, is impossible improve runtime. Maybe the problem is due to video resolution, if W and H size is long, then buffer reading and showing video frames will be more delayed. A simple solution is decrease the resolution of videos before of reading. Another solution is create an second thread to only video acquisition. Another solution is to read video using C++. I have compared the performance and it is about 3 times faster. — Roger Figueroa Quintero, Oct 09 '19 at 06:33
Also note that your problem could be hardware: memory and data-bus bandwidth, and processor speed. — Roger Figueroa Quintero, Oct 09 '19 at 06:41
use a faster device (more processing power) or use hardware acceleration or optimized code to receive, decode and display the stream. OpenCV might not be the fastest library for receiving, decoding and rendering (but it's very easy to use) — Micka, Oct 15 '19 at 07:40

nathancy · Accepted Answer · 2022-05-06T03:35:11.053

My hypothesis is that the jitter is most likely due to network limitations and occurs when a frame packet is dropped. When a frame is dropped, this causes the program to display the last "good" frame which results in the display freezing. This is probably a hardware or bandwidth issue but we can alleviate some of this with software. Here are some possible changes:

1. Set maximum buffer size

We set the cv2.videoCapture() object to have a limited buffer size with the cv2.CAP_PROP_BUFFERSIZE parameter. The idea is that by limiting the buffer, we will always have the latest frame. This can also help to alleviate the problem of frames randomly jumping ahead.

2. Set frame retrieval delay

Currently, I believe the read() is reading too fast even though it is in its own dedicated thread. This may be one reason why all the frames appear to pool up and suddenly burst in the next frame. For instance, say in a one second time interval, it may produce 15 new frames but in the next one second interval, only 3 frames are returned. This may be due to the network packet frame loss so to ensure that we obtain constant frame rates, we simply add a delay in the frame retrieval thread. A delay to obtain roughly ~30 FPS does a good job to "normalize" the frame rate and smooth the transition between frames incase there is packet loss.

Note: We should try to match the frame rate of the stream but I'm not sure what the FPS of the webcam is so I just guessed 30 FPS. Also, there is usually a "direct" stream link instead of going through a intermediate webserver which can greatly improve performance.

If you try using a saved .mp4 video file, you will notice that there is no jitter. This confirms my suspicion that the problem is most likely due to network latency.

from threading import Thread
import cv2, time

class ThreadedCamera(object):
    def __init__(self, src=0):
        self.capture = cv2.VideoCapture(src)
        self.capture.set(cv2.CAP_PROP_BUFFERSIZE, 2)
       
        # FPS = 1/X
        # X = desired FPS
        self.FPS = 1/30
        self.FPS_MS = int(self.FPS * 1000)
        
        # Start frame retrieval thread
        self.thread = Thread(target=self.update, args=())
        self.thread.daemon = True
        self.thread.start()
        
    def update(self):
        while True:
            if self.capture.isOpened():
                (self.status, self.frame) = self.capture.read()
            time.sleep(self.FPS)
            
    def show_frame(self):
        cv2.imshow('frame', self.frame)
        cv2.waitKey(self.FPS_MS)

if __name__ == '__main__':
    src = 'https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8'
    threaded_camera = ThreadedCamera(src)
    while True:
        try:
            threaded_camera.show_frame()
        except AttributeError:
            pass

Related camera/IP/RTSP/streaming, FPS, video, threading, and multiprocessing posts

Can you tell me why the waiting (FPS syncing) is needed? Isn't the point of the buffer to capture frames as they come? It feels like it wouldn't matter what is grabbed from the top of the buffer. Building an artificial delay feels counterintuitive for me. Would that be about resource sharing in threads or something? Or is it that the images don't come in real-time exactly, but as they're able to be reconstructed or re-delivered. Either way, thank you for the detailed response. This works very well. — Conic, Oct 15 '19 at 18:18
@Conic The FPS syncing is needed because generally you want to sync with the source. If you are polling too slow, you will have periods where you don't have a new frame so you will be displaying the old frame. If you poll too fast (like before), you will display all the frames instantly. In both cases, this may cause the jitter/freeze so by inserting a delay, you can even out the frame distribution as it is displayed. Since there are two separate threads, once a frame is received, it is immediately displayed so that's another reason why we need a delay. — nathancy, Oct 15 '19 at 20:13
Here's an analogy, say you're watching a Youtube video on 2x speed but you are using dial-up internet. The buffer rate will load slower than the video actually playing so eventually, the frames will catch up to the buffer and the video will freeze. This is what happens when the FPS is not synced. If we revert back to 1x speed, then the frames will be evened out and match the speed of the buffer rate so we don't get any freezing/jitter. The video then looks smooth — nathancy, Oct 15 '19 at 20:15
OpenCV VideoCapture isn't polling at all. could you explain what you mean? — Christoph Rackwitz, Dec 11 '20 at 18:25
@ChristophRackwitz actually it's polling in the `update()` which is called inside started thread: `Thread(target=self.update, args=())` — sur0k, Apr 09 '21 at 17:49
It's constantly polling. The entire operation of capturing frames is put inside the thread so that if the program is exited, the capture operation will not hang as a zombie process. If we don't set `daemon` to `True`, it will be a zombie process when your main program dies. We set daemon `True` so that when our parent process dies, this will also kill all child processes. See [threading documentation](https://docs.python.org/3/library/threading.html#thread-objects) for more information — nathancy, Aug 10 '21 at 08:14
Could you please explain to me why you used a threaded camera. I set a maximum buffer size and a frame retrieval delay as you suggested and that was enough to get a smooth streaming. — Ben Zayed, Nov 15 '21 at 15:37
@BenZayed For a single camera just setting buffer size and delay should work but if you have multiple cameras running simultaneously you may need to use threading (I use this technique with 8 simultaneous camera streams) — nathancy, Nov 17 '21 at 01:29

score 7 · Answer 2 · edited Jun 20 '20 at 09:12

Attempt at threading

I've attempted this solution from nathancy with minor success.

It involves:

creating a separate thread for image capture from the source
using the main thread exclusively for display.

Code:

import cv2
from threading import Thread

class ThreadedCamera(object):
    def __init__(self, source = 0):

        self.capture = cv2.VideoCapture(source)

        self.thread = Thread(target = self.update, args = ())
        self.thread.daemon = True
        self.thread.start()

        self.status = False
        self.frame  = None

    def update(self):
        while True:
            if self.capture.isOpened():
                (self.status, self.frame) = self.capture.read()

    def grab_frame(self):
        if self.status:
            return self.frame
        return None

if __name__ == '__main__':
    stream_link = "https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8"
    streamer = ThreadedCamera(stream_link)

    while True:
        frame = streamer.grab_frame()
        if frame is not None:
            cv2.imshow("Context", frame)
        cv2.waitKey(1)

Jittery, but real-time results

.

The streaming works. It maintains real-time. However, it is as if all the frames pool up and suddenly burst into the video. I would like somebody to explain that.

Room for improvement

The real-time stream can be found here.

https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet

This site is scraped for the m3u8 using python's streamlink stream scraper.


import streamlink

streams = streamlink.streams("https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet")
print(streams)

which yeilds:

OrderedDict([

('720p',<HLSStream('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w202109066.m3u8')>),

('live', <RTMPStream({'rtmp': 'rtmp://videos3.earthcam.com/fecnetwork/', 'playpath': '9974.flv', 'pageUrl': 'https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet','swfUrl': 'http://static.earthcam.com/swf/streaming/stream_viewer_v3.swf', 'live': 'true'}, redirect=False>),

('worst', <HLSStream('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w202109066.m3u8')>),

('best', <RTMPStream({'rtmp': 'rtmp://videos3.earthcam.com/fecnetwork/', 'playpath': '9974.flv', 'pageUrl': 'https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet', 'swfUrl': 'http://static.earthcam.com/swf/streaming/stream_viewer_v3.swf', 'live': 'true'}, redirect=False>)

])

The possibility that the streams are being read wrong.

Yes I did. It's the answer with the green checkmark next to it from nathancy. — Conic, Jul 30 '21 at 14:08
Oops sorry. Since you said nathancy's answer was a minor success. I thought you meant the answer to this question (didnt check the link). Anyway. Thanks. — Arunava, Jul 30 '21 at 14:12
I can see how that might be misleading. I was referring to a different solution of his from another stack overflow post. He then responded with an updated answer that did FPS syncing. That ended up being the solution that worked for me. — Conic, Jul 30 '21 at 14:14

score 0 · Answer 3 · answered Jan 06 '23 at 19:28

I would suggest double checking the compatible video stream codecs with the hardware. I ran into the same issue, frame rate dropped to 5 fps only during streaming, because it was defaulting to a format that is not being streamed so it would convert it then display very lagged (~1s) with lower fps as well.

use Self.capture.set(cv2.CAP_PROP_FOURCC ,cv2.VideoWriter_fourcc('M', 'J', 'P', 'G') ) with the proper codec in place of MJPG and with your cv2.VideoCapture and see if that helps.