0

I was able to receive/view UDP h264 packets through VLC command line (i.e. VLC --network-caching 0 --demux h264 udp://...)

I am planning on processing those frames through OpenCV algorithms. However, I can't seem to find a way to send the VLC frames over to my Python OpenCV script.

Is it possible to pipe VLC stream output to be processed through Numpy in a separate script?

I have previously tried directly streaming to OpenCV by using its VideoCapture function, but for some reason, the video stalls and stops at a distorted black image. For now, it seems like incorporating VLC is the only solution (even though I am not totally sure why VLC works over other methods).

Thank you.

EDIT:

The following is a snippet of the error message on the terminal. It seems like there are problems with the first few frames, but I don't know why the stream works on VLC. From the client, I first sent a default key frame data, and then sent video feed h264 data.

[h264 @ 0x7f9c50020200] top block unavailable for requested intra mode -1
[h264 @ 0x7f9c50020200] error while decoding MB 7 0, bytestream 7208
[h264 @ 0x7f9c50020200] top block unavailable for requested intra mode -1
[h264 @ 0x7f9c50020200] error while decoding MB 8 9, bytestream 7381
J. S.
  • 89
  • 1
  • 3
  • 9
  • What is your streaming server? – zindarod Jun 18 '18 at 20:26
  • I am sending h264 data from an Android phone (acting as a client), which my computer receives through VLC command. – J. S. Jun 18 '18 at 20:34
  • what video format do you want to use in OpenCV? `vlc` or `ffmpeg` can be used to change the video encoding format. – Ereli Jun 18 '18 at 21:08
  • I've only worked with image formats for OpenCV, so I'm not really sure about how the video format would be used on OpenCV. Since it has to be real-time extraction of frames, I assumed the h264 data will be decoded into image frames, instead of through a video format. – J. S. Jun 18 '18 at 22:01
  • vlc uses live555 which is open source. You could write your own live555 client and wrap it in a C library. Although I'm interested in whether there is a way to use vlc programmatically, too :) – Micka Jun 18 '18 at 22:50
  • 1
    @Micka VLC is built around [libVLC](https://wiki.videolan.org/LibVLC/). Documentation [here](https://forum.videolan.org/viewtopic.php?t=76751). – zindarod Jun 19 '18 at 13:59

1 Answers1

9

You can use ffmpeg for streaming.

First test ffmpeg streaming in terminal. In linux, we use v4l2 to grab frames from camera.

Server:

ffmpeg -f v4l2 -i /dev/video0 -preset ultrafast -vcodec libx264 -tune zerolatency -b 900k -f h264 udp://127.0.0.1:5000

Client:

ffplay udp://127.0.0.1:5000

If you're able to view the stream on the client side, then we can use OpenCV for image processing. OpenCV must have ffmepg support. See this link for ffmpeg support check.

    cap = cv2.VideoCapture('udp://127.0.0.1:5000',cv2.CAP_FFMPEG)
    if not cap.isOpened():
        print('VideoCapture not opened')
        exit(-1)

    while True:
        ret, frame = cap.read()

        if not ret:
            print('frame empty')
            break

        cv2.imshow('image', frame)

        if cv2.waitKey(1)&0XFF == ord('q'):
            break

    cap.release()
    cv2.destroyAllWindows()
zindarod
  • 6,328
  • 3
  • 30
  • 58
  • Thank you for the response. I have tried running ffplay command but the terminal says the command is not found. I have not tested VideoCapture with cv2.CAP_FFMPEG yet, but would it be possible that this is the reason why the OpenCV video capture did not work? I have tried googling on adding ffplay onto the ffmpeg installation, but I have not been successful as of now. – J. S. Jun 18 '18 at 22:08
  • 1
    @J.S. You don't need `ffplay` to view stream. You can write it to file by: `ffmpeg -i udp://127.0.0.1:5000 ./out.mp4`. – zindarod Jun 18 '18 at 22:33
  • 1
    When I tried with the bash command and VideoCapture with cv2.CAP_FFMPEG, they both stop at a distorted black image and crashes (same as what I described in the original post). I am still confused about why I can stream on VLC but not on FFMPEG. I have updated the original post with the error messages. – J. S. Jun 19 '18 at 15:04
  • What about be the -f flag value for streaming from UDP? Also, why do you do ffmpeg on the Server and then ffplay on the Client? – AlvinfromDiaspar Feb 21 '20 at 00:36
  • Also, what version of OpenCV are you using? – AlvinfromDiaspar Feb 21 '20 at 00:37