0

I have a scenario where I need to retrieve a video stream from an RTMP server, apply image processing (specifically, adding blur to frames), and then forward the processed stream to another RTMP server (in this case, Twitch).

Currently, I'm using ffmpeg in conjunction with cv2 to retrieve and process the stream. However, this approach introduces significant lag when applying the blur. I'm seeking an alternative method that can achieve the desired result more efficiently. I did attempt to solely rely on ffmpeg for the entire process, but I couldn't find a way to selectively process frames based on a given condition and subsequently transmit only those processed frames.

Is there a more efficient approach or alternative solution that can address this issue and enable real-time video stream processing with minimal lag?

Thanks in advance!

def forward_stream(server_url, stream_key, twitch_stream_key):
    get_ffmpeg_command = [...]

    send_ffmpeg_command [...]

    # Start get FFmpeg process
    read_process = subprocess.Popen(get_ffmpeg_command, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)

    # Start send FFmpeg process
    send_process = send_process = subprocess.Popen(send_ffmpeg_command, stdin=subprocess.PIPE, stderr=subprocess.DEVNULL)

    # Open video capture
    cap = cv2.VideoCapture(f'{server_url}')

    while True:
        # Read the frame
        ret, frame = cap.read()
        if ret:
            # Apply machine learning algorithm
            should_blur = machine_learning_algorithm(frame)

            # Apply blur if necessary
            if machine_learning_algorithm(frame):
                frame = cv2.blur(frame, (25, 25))

            # Write the frame to FFmpeg process
            send_process.stdin.write(frame.tobytes())
        else:
            break

    # Release resources
    cap.release()
    read_process.stdin.close()
    read_process.wait()
Christoph Rackwitz
  • 11,317
  • 4
  • 27
  • 36
dumbQuestions
  • 45
  • 3
  • I don't think there is an approach that is truly different. The described process, requires decoding, processing and re-encoding. Even for modifying few frames, we have to decode and re-encode. For reducing the "lag" (latency), we may adjust the arguments of the decoder and the encoder. I see you are not using `read_process`... for adjusting the arguments use FFmpeg instead of `cv2.VideoCapture`. Reducing the latency of the encoder to minimum may reduce the compression ratio and increase the bandwidth and/or reduce the video quality (I don't know if it's going to be a practical solution). – Rotem May 19 '23 at 09:50
  • using a subprocess is inefficient. use PyAV. -- `read_process` is never actually used, or is it? – Christoph Rackwitz May 19 '23 at 09:59
  • @ChristophRackwitz From latency perspective, using subprocess with pipes versus PyAV (for encoding/decoding) is about the same. Reading and writing from pipes has some overhead compared to PyAV, but I don't think it is very significant. Preserving the audio may be a good reason for using PyAV. – Rotem May 19 '23 at 10:55
  • preserving all the other metadata (timestamps) would be a good idea too – Christoph Rackwitz May 19 '23 at 10:57
  • @Rotem Hey, thanks for the answer! Do you know how I could use FFmpeg instead of cv2.VideoCapture? I don't know how I can get the frames from FFmpeg, process them, and transmit them to the `send_process`? – dumbQuestions May 21 '23 at 17:17
  • The main principle is writing frames to `stdin` pipe. You may take a look at my [following answer](https://stackoverflow.com/a/61281547/4926757) for example. For reducing the latency of the encoder process, try adding `-tune zerolatency` output argument. For reducing the latency of the decoder process, try adding `-flags low_delay -probesize 32` input arguments. Start the latency test with low resolution video. – Rotem May 21 '23 at 18:29

0 Answers0