Stream RAW8 video from camera using openCV and Python [Windows]

Question

We have a camera that streams RAW8 video at 1920x1080. The GUID used is GREY. We are able to stream video from this camera using ffmpeg on Windows with the below command:

ffmpeg -f dshow -pix_fmt gray -video_size 1920x1080 -i video="CAM0" -f nut - | ffplay -

We are now trying to grab images from this camera using OpenCV using the below code snippet, but its unable to grab any frame (frame_grabbed is always false)

import cv2
import numpy as np

# reading the video from CAM0
source = cv2.VideoCapture(1)

height = 1920
width = 1080

source.set(cv2.CAP_PROP_FRAME_WIDTH, width)
source.set(cv2.CAP_PROP_FRAME_HEIGHT, height)


image = np.zeros([height, width, 3], np.uint8)

while True:
    # Extracting the frames
    frame_grabbed , image = source.read()

    if (frame_grabbed ):
        colour1 = cv2.cvtColor(image, cv2.COLOR_BayerRG2BGR)
        cv2.imshow("Demosaiced image", colour1)
    else:
        print("No images grabbed")

#Exit on q
    key = cv2.waitKey(1)
    if key == ord("q"):
       break

# closing the window
cv2.destroyAllWindows()
source.release()

Are we missing something here?

We then came across this post to pipe ffmpeg output to python (link). However, when we are passing the command as below:

command = [ 'ffmpeg.exe',
            '-f', 'dshow',
            '-i', 'video="CAM0"',
            '-pix_fmt', 'gray',
            '-video_size','1920x1080'
            '-f', 'nut', '-']

its throwing

Could not find video device with name ["CAM0"] among source devices of type video. video="CAM0": I/O error

I have verified that the camera is present using the below command:

command = [ 'ffmpeg.exe',
            '-list_devices', 'true',
            '-f', 'dshow',
            '-i', 'dummy']

This detects CAM0 as shown below:

ffmpeg version 5.0.1-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 11.2.0 (Rev7, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-liblensfun --enable-libvidstab 
--enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
  libpostproc    56.  3.100 / 56.  3.100
[dshow @ 000001ea39e40600] "HP HD Camera" (video)
[dshow @ 000001ea39e40600]   Alternative name "@device_pnp_\\?\usb#vid_04f2&pid_b6bf&mi_00#6&1737142c&0&0000#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\global"
[dshow @ 000001ea39e40600] "CAM0" (video)
[dshow @ 000001ea39e40600]   Alternative name "@device_pnp_\\?\usb#vid_0400&pid_0011&mi_00#7&1affbd5b&0&0000#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\global"

In short, we are able to capture video using ffmpeg commandline, but unable to grab any frame using OpenCV videocapture or ffmpeg in opencv. Any pointers?

Thanks!

Answer 1

I don't know why reading the frames using OpenCV is not working, but we may use FFmpeg CLI instead.

• Execute FFmpeg as sub-process, and set the output to stdout pipe.
• Read the raw video frames from stdout pipe, and convert each frame to 1920x1080 uint8 NumPy array.

---

Code sample:

import cv2
import numpy as np
import subprocess as sp

width = 1920
height = 1080

ffmpeg_command = ['ffmpeg',  # Make sure ffmpeg.exe is in the execution path
                  '-f', 'dshow',  # The input format is dshow (camera)
                  '-video_size', f'{width}x{height}',  # input resolution is 1920x1080
                  '-pixel_format', 'gray',  # input pixel format is gray
                  '-i', 'video=CAM0',  # May also try 'video="CAM0"'
                  '-f', 'rawvideo',  # The output format is rawvideo
                  '-pix_fmt', 'gray',  # The output pixel format is gray
                  'pipe:']  # Output the video to stdout pipe

# Open sub-process that gets in_stream as input and uses stdout as an output PIPE.
ffmpeg_process = sp.Popen(ffmpeg_command, stdout=sp.PIPE)

while True:
    # Read width*height bytes from stdout (1 frame)
    raw_frame = ffmpeg_process.stdout.read(width*height)

    if len(raw_frame) != (width*height):
        print('Error reading frame!!!')  # Break the loop in case of an error (too few bytes were read).
        break

    # Convert the bytes read into a NumPy array, and reshape it to video frame dimensions
    frame = np.frombuffer(raw_frame, np.uint8).reshape((height, width))

    # Show the video frame
    cv2.imshow('frame', frame)
    key = cv2.waitKey(1)
    if key == ord("q"):
        break

ffmpeg_process.stdout.close();  # Closing stdout pipe also terminates FFmpeg sub-process.
cv2.destroyAllWindows()

Answer 2

Try using 'rawvide' as the output format. You may need to move all your camera options to the left of the -i argument.

You can try my ffmpegio package if you need a quick and dirty way:

import ffmpegio

show_log = True

# if you have to record for a fixed duration 
t_record = 5
fs,F = ffmpegio.video.read('video=CAM0',f_in='dshow',
                           'pix_fmt='gray', video_size_in='1920x1080', t=5,
                           show_log=show_log)

# if you need to stream input
with ffmpegio.open('video=CAM0','rv', f_in='dshow',
                   'pix_fmt_in='gray', video_size_in='1920x1080',
                   show_log=show_log) as cam:
    for frame in cam:
        # frame = np.array of shape (1x)1920x1080x1 (I think, reshape as needed)

If you need help on this, feel free to create a thread on the GitHub discussion board (not to clutter up here).