Get each video's frames as an QImage

Question

I'm starting a project that consists of receive video from an RTSP server and showing it in a panel using QT c++ and GSTreamer. I want to receive every frame as a QImage object to run some features that I need. I know that we can use QTMultimedia to link the Gstreamer and view the video, but in this particular case, I would like to have QImages. I found a way (or i think i did) to get the samples from the video by calling the function "gst_app_sink_pull_sample". We get a GSTSample from it. However, i did not find any way to convert this into, for example, a JPG raw data which is easy to convert into a QImage.

I also found a way to access the data from GSTSample from here: "How to get video stream frame-by-frame from Gstreamer pipeline? (without OpenCV)" but, once again, i've no idea how can i convert this data into a QImage.

 /* Initialize GStreamer */
        gst_init (NULL, NULL);

        /* Create the elements */
        GstElement * source = gst_element_factory_make ("videotestsrc", "source"); // HERE I'M USING THE GSTREAMER SOURCE TEST
        GstElement * sink = gst_element_factory_make ("appsink", "sink");
        GstAppSink *appsink = GST_APP_SINK(sink);

        /* Create the empty pipeline */
         GstElement * pipeline = gst_pipeline_new ("test-pipeline");

        if (!pipeline || !source || !sink) {
            g_printerr ("Not all elements could be created.\n");
            return ;
        }

        /* Build the pipeline */
        gst_bin_add_many (GST_BIN (pipeline), source, sink, NULL);
        if (gst_element_link (source, sink) != TRUE) {
            g_printerr ("Elements could not be linked.\n");
            gst_object_unref (pipeline);
            return ;
        }

        /* Modify the source's properties */
        g_object_set (source, "pattern", 0, NULL);


        /* Start playing */
        ret = gst_element_set_state (pipeline, GST_STATE_PLAYING);
        if (ret == GST_STATE_CHANGE_FAILURE) {
            g_printerr ("Unable to set the pipeline to the playing state.\n");
            gst_object_unref (pipeline);
            return;
        }


        // My question starts HERE
        GstSample *sample = nullptr;

        do{

            sample = gst_app_sink_pull_sample(appsink); // Get the frame
            if (!sample) {
                printf("sample is NULL\n");
            }else{
                
                // Get the raw data (or trying to...)
                GstBuffer * buffer = gst_sample_get_buffer(sample);
                GstMapInfo info;
                gst_buffer_map(buffer, &info, GST_MAP_READ);

                // Dumb way to check if the frame is being received properly
                QImage imageAux((const unsigned char*)info.data, 100, 100, QImage::Format_RGB16);
                imageAux.save("out.jpg"); //returns garbage

            }


        }while(sample);

I want to propose an alternative solution for which I need more information about your requirement: From what I understand you want to get each frame, convert it to QImage, modify it, but after that where do you want to place it? Do you want it to be displayed in the window with the modifications or do you want to do something else with it? — eyllanesc, Sep 16 '21 at 17:28
Hello @eyllanesc, yes i want to display it in a windows form. — Daniel Lopes, Sep 17 '21 at 09:04

score 0 · Answer 1 · answered Sep 17 '21 at 21:29

According to that topic

cv::Mat frame(cv::Size(width, height), CV_8UC4, (char *)map.data, cv::Mat::AUTOSTEP);

The actual image data begins from map.data pointer. And have CV_8UC4 format in terms of OpenCV. Construct the QImage with the next constructor QImage(uchar *data, int width, int height, int bytesPerLine, QImage::Format format, ImageCleanupFunction cleanupFunction = nullptr, void *cleanupInfo = nullptr)

You should understand what is CV_8UC4 format. And find appropriate one among QImage::Format list. And find other parameters in that 'map' to push into the constructor. In good case, when you found appropriate format in QImage::Format, push the map.data into uchar *data. In wrong case you should try to find another way to convert CV_8UC4 into something suitable with Qt's images.

Also, the author of the mentioned topic might mistake with CV_8UC4. Then there is the issue what kind of image format under map.data pointer. But the steps to get QImage can be suitable anyway, once you get the right format.

Thank you! I will test it soon. I reply to you as soon as i've news. — Daniel Lopes, Sep 20 '21 at 16:57

score 0 · Answer 2 · edited Sep 25 '21 at 21:16

Result from cv::imwrite

The result is not good. The image above results from saving the Mat frame() as image using cv::imwrite("some.jpg", frame); It may return an image like that: videotestsrc image

There is the code:

GstBuffer * buffer = gst_sample_get_buffer(sample);
GstMapInfo map;
GstCaps *caps = gst_sample_get_caps(sample);
GstStructure *structure = gst_caps_get_structure(caps, 0);
const int width = g_value_get_int(gst_structure_get_value(structure, "width"));
const int height = g_value_get_int(gst_structure_get_value(structure, "height"));

gst_buffer_map(buffer, &map, GST_MAP_READ);

cv::Mat frame(cv::Size(width, height),  CV_8UC3, (char *)map.data, cv::Mat::AUTO_STEP);

cv::imwrite("some.jpg", frame);

QImage imgIn= QImage((uchar*) frame.data, frame.cols, frame.rows, frame.step, QImage::Format_RGB888);
imgIn.save("out.jpg");

The QImage is quite similar, but now I'm more focused on trying to understand what is happening with v::imwrite("some.jpg", frame); I did try several frames formats like CV_8UC4, CV_8UC1, etc...

score 0 · Answer 3 · edited Sep 25 '21 at 21:16

Got it! I found this example: Qt+GStreamer: How to take a snapshot while playing live video stream It uses the QT-Gstreamer libraries which unfortunately is not working in my system. But after some modifications in order to use Gstreamer only, I could get a QImage. I was missing the format of the frame. So, I need to convert it into a known format (gst_caps_new_simple(...) ) and finally convert it into a QImage.

/* Initialize GStreamer */
gst_init (NULL, NULL);

/* Create the elements */
source = gst_element_factory_make ("videotestsrc", "source");
//sink = gst_element_factory_make ("autovideosink", "sink");
sink = gst_element_factory_make ("appsink", "sink");
GstAppSink *appsink = GST_APP_SINK(sink);

/* Create the empty pipeline */
pipeline = gst_pipeline_new ("test-pipeline");

if (!pipeline || !source || !sink) {
    g_printerr ("Not all elements could be created.\n");
    return ;
}

/* Build the pipeline */
gst_bin_add_many (GST_BIN (pipeline), source, sink, NULL);
if (gst_element_link (source, sink) != TRUE) {
    g_printerr ("Elements could not be linked.\n");
    gst_object_unref (pipeline);
    return ;
}

/* Modify the source's properties */
g_object_set (source, "pattern", 0, NULL);


// Start the ThreadVideoProcessor

/* Start playing */
ret = gst_element_set_state (pipeline, GST_STATE_PLAYING);
if (ret == GST_STATE_CHANGE_FAILURE) {
    g_printerr ("Unable to set the pipeline to the playing state.\n");
    gst_object_unref (pipeline);
    return;
}

GstSample *sample = nullptr;

do{

    sample = gst_app_sink_pull_sample(appsink);
    if (!sample) {
        printf("sample is NULL\n");
    }else{

        GstMapInfo map;
        GstCaps *caps = gst_sample_get_caps(sample);
        GError *err = NULL;

        GstStructure *structure = gst_caps_get_structure(caps, 0);
        const int width = g_value_get_int(gst_structure_get_value(structure, "width"));
        const int height = g_value_get_int(gst_structure_get_value(structure, "height"));

        GstCaps * capsTo = gst_caps_new_simple("video/x-raw",
                                               "format", G_TYPE_STRING, "RGB",
                                               "width", G_TYPE_INT, width,
                                               "height", G_TYPE_INT, height,
                                               NULL);


         GstSample *convertedSample = gst_video_convert_sample(sample,capsTo,GST_SECOND,&err);


        if (convertedSample == nullptr) {
            //qWarning() << "gst_video_convert_sample Failed:" << err->message;
        }
        else {
            //qDebug() << "Converted sample caps:" << convertedSample->caps()->toString();

            GstBuffer * buffer = gst_sample_get_buffer(convertedSample);
            gst_buffer_map(buffer, &map, GST_MAP_READ);

            QImage snapShot = QImage((const uchar *)map.data,
                              width,
                              height,
                              QImage::Format_RGB888);

            //qDebug() << "Saving snap to" << "out.jpg";
            snapShot.save("out.jpg");

    }

   }

After execute it:

enter image description here

Get each video's frames as an QImage

3 Answers3