Unable to route webcam video to virtual video device on Linux (via OpenCV)

Question

I would like to put a video overlay onto an incoming webcam stream via OpenCV. As a first step I'm trying to stream the incoming video from a webcam at /dev/video0 to a virtual video device at /dev/video1 (v4l2loopback). Unfortunately I'm not able to forward the webcam stream to the v4l2loopback device.

I found this thread: How to write/pipe to a virtual webcam created by V4L2loopback module? but the links there didn't really help me.

Getting and watching the stream from the webcam works well with some little demo code from the OpenCV page. As well the example code from the v4l2loopback page, for e.g. playing a static video file out to /dev/video1 works well. I can watch the video when I connect VLC to /dev/video1

I already read that the v4l2loopback device must be controlled via regular Linux driver commands (open, ioctl, write etc.). So there is no wrapper class from within OpenCV to write to the loopback device. My webcam streams with 640x480 and as MJPG. The interesting thing is that I can connect to the loopback device with VLC and I can see the correct resolution, Codec and FPS displayed when I press play. The time counter on the progress bar also starts running. But the screen remains black (with VLC Logo).

What I'm roughly doing is this (BTW: the code might not compile ... didn't want clutter everything here ... let me know if you need more details):

int main ( int argc, char **argv ) {
    cv::VideoCapture cap;
    struct v4l2_format vid_format;
    size_t framesize = 640 * 480 * 3; // 3 Bytes per  pixel
    __u8 *buffer = null;
    int fd = null;

    cap.open  ( "/dev/video0" );
    fd = open ( "/dev/video1", O_RDWR );

    memset ( &vid_format, 0, sizeof(vid_format) );
    vid_format.type                = V4L2_BUF_TYPE_VIDEO_OUTPUT;
    vid_format.fmt.pix.width       = cap.get(CV_CAP_PROP_FRAME_WIDTH);
    vid_format.fmt.pix.height      = cap.get(CV_CAP_PROP_FRAME_HEIGHT);
    vid_format.fmt.pix.pixelformat = { 'M', 'J', 'P', 'G' };
    vid_format.fmt.pix.sizeimage   = framesize;
    vid_format.fmt.pix.field       = V4L2_FIELD_NONE;
    vid_format.fmt.pix.colorspace  = V4L2_COLORSPACE_SRGB;
    ioctl ( fd, VIDIOC_S_FMT, &vid_format );

    buffer = (__u8*) malloc ( sizeof(__u8) *framesize );
    memset ( buffer, 0, framesize );

    for(;;) {
        cv::Mat frame;
        cap >> frame;
        write ( fd, &frame.data, framesize );
    }
}

It would be really great if somebody could give me a hint how I need to convert the wecam data in order to get it accepted by VLC.

Answer 1

After a lot of researching, I finally was able to develop a working solution. There are a lot of steps which need to be performed and which I will discuss in detail below:

General

As described in my question above, the goal is to be able to take the incoming stream of a webcam and forward it to a virtual video device, which in turn can be opened with tools like VLC. This is considered to be a first step in order to be able to do further image manipulation.

1) v4l2loopback

v4l2loopback is a virtual video device (kernel module) for linux. Sources can be downloaded from here https://github.com/umlaeute/v4l2loopback. After downloading, the following steps must be performed in order to run it:

make
sudo make install
sudo depmod -a
sudo modprobe v4l2loopback

If you want to use this video device in Chrome (WebRTC) you need to execute the last line with an additional parameter:

sudo modprobe v4l2loopback exclusive_caps=1

Note that exlusive_caps is an array, so if the above doesn't work, try:

sudo modprobe v4l2loopback exclusive_caps=1,1,1,1,1,1,1,1

Information: It is important to note that the v4l2loopback device must be set to the same resolution like the resolution you want to use in the sample below. I have set the defines in the sample to FullHD as you can see. If you want e.g. 800x600, you either need to change the default in the v4l2loopback code before compilation or change the resolution, when inserting the module, via the additional cmd line parameters max_width and max_height. The kernel module operates by default with a resolution of 640x480. You can get more details and all supported parameters by using:

modinfo v4l2loopback

2) OpenCV

OpenCV is a library which supports capturing and live video manipulation. For building OpenCV please got to this page http://docs.opencv.org/3.0-beta/doc/tutorials/introduction/linux_install/linux_install.html which explains all steps in detail.

3) Sample code

You can build/run the sample code below in the following way:

g++ -ggdb `pkg-config --cflags --libs opencv` sample.cpp -o sample
./sample

Here is the code:

#include <stdio.h>
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <linux/videodev2.h>
#include "opencv2/opencv.hpp"

#define VIDEO_OUT "/dev/video0" // V4L2 Loopack
#define VIDEO_IN  "/dev/video1" // Webcam

#define WIDTH  1920
#define HEIGHT 1080


int main ( int argc, char **argv ) {
    cv::VideoCapture cap;
    struct v4l2_format vid_format;
    size_t framesize = WIDTH * HEIGHT * 3;
    int fd = 0;

    if( cap.open ( VIDEO_IN ) ) {
        cap.set ( cv::CAP_PROP_FRAME_WIDTH , WIDTH  );
        cap.set ( cv::CAP_PROP_FRAME_HEIGHT, HEIGHT );
    } else {
        std::cout << "Unable to open video input!" << std::endl;
    }

    if ( (fd = open ( VIDEO_OUT, O_RDWR )) == -1 )
        printf ("Unable to open video output!");

    memset ( &vid_format, 0, sizeof(vid_format) );
    vid_format.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;

    if ( ioctl ( fd, VIDIOC_G_FMT, &vid_format ) == -1 )
        printf ( "Unable to get video format data. Errro: %d\n", errno );

    vid_format.fmt.pix.width       = cap.get ( CV_CAP_PROP_FRAME_WIDTH  );
    vid_format.fmt.pix.height      = cap.get ( CV_CAP_PROP_FRAME_HEIGHT );
    vid_format.fmt.pix.pixelformat = V4L2_PIX_FMT_RGB24;
    vid_format.fmt.pix.sizeimage   = framesize;
    vid_format.fmt.pix.field       = V4L2_FIELD_NONE;

    if ( ioctl ( fd, VIDIOC_S_FMT, &vid_format ) == -1 )
        printf ( "Unable to set video format! Errno: %d\n", errno );

    cv::Mat frame ( cap.get(CV_CAP_PROP_FRAME_HEIGHT), 
    cap.get(CV_CAP_PROP_FRAME_WIDTH), CV_8UC3 );
  
    printf ( "Please open the virtual video device (/dev/video<x>) e.g. with VLC\n" );

    while (1) {
        cap >> frame;
        cv::cvtColor ( frame, frame, cv::COLOR_BGR2RGB ); // Webcams sometimes deliver video in BGR not RGB. so we need to convert
        write ( fd, frame.data, framesize );
    }
}