Combining videos in opencv and Qt

Question

I am trying to write a program that will combine 4 videos into a single video, arranging them into a 2x2 grid. Below is the relevant code:

void CombineVideoApp::on_GoButton_clicked()
{
    std::string Video1Name = Video1Path.toUtf8().constData();
    std::string Video2Name = Video2Path.toUtf8().constData();
    std::string Video3Name = Video3Path.toUtf8().constData();
    std::string Video4Name = Video4Path.toUtf8().constData();

    int VideoWidth = 640;
    int VideoHeight = 360;

    cv::Mat Video1Frame, Video2Frame, Video3Frame, Video4Frame; // Creating Mat objects to store the captured frames
    cv::Mat Video1Shrunk, Video2Shrunk, Video3Shrunk, Video4Shrunk; //Creating Mat objects to store the shrunk frames
    cv::Mat FullCombinedVideo(VideoWidth,VideoHeight, Video1Shrunk.type()); //Creating Mat object to store the combined video
    cv::Mat CombinedVideoTop(VideoWidth, VideoHeight/2, Video1Shrunk.type());
    cv::Mat CombinedVideoBottom(VideoWidth, VideoHeight/2, Video1Shrunk.type());


    cv::VideoCapture Video1(Video1Name);
    cv::VideoCapture Video2(Video2Name);
    cv::VideoCapture Video3(Video3Name);
    cv::VideoCapture Video4(Video4Name);



    double Video1FrameCount = Video1.get(CV_CAP_PROP_FRAME_COUNT);
    double Video2FrameCount = Video2.get(CV_CAP_PROP_FRAME_COUNT);
    double Video3FrameCount = Video3.get(CV_CAP_PROP_FRAME_COUNT);
    double Video4FrameCount = Video4.get(CV_CAP_PROP_FRAME_COUNT);

    double CombinedFrameTopCount = min(Video1FrameCount, Video2FrameCount);
    double CombinedFrameBottomCount = min(Video3FrameCount, Video4FrameCount);
    double CombinedFrameCount = min(CombinedFrameBottomCount, CombinedFrameTopCount);

    Video1.set(CV_CAP_PROP_FRAME_COUNT, CombinedFrameCount);
    Video2.set(CV_CAP_PROP_FRAME_COUNT, CombinedFrameCount);
    Video3.set(CV_CAP_PROP_FRAME_COUNT, CombinedFrameCount);
    Video4.set(CV_CAP_PROP_FRAME_COUNT, CombinedFrameCount);


    for(;;)
    {
        Video1 >> Video1Frame;
        Video2 >> Video2Frame;
        Video3 >> Video3Frame;
        Video4 >> Video4Frame;

        cv::resize(Video1Frame, Video1Shrunk, Size((VideoWidth/2), (VideoHeight/2)));
        cv::resize(Video2Frame, Video2Shrunk, Size((VideoWidth/2), (VideoHeight/2)));
        cv::resize(Video3Frame, Video3Shrunk, Size((VideoWidth/2), (VideoHeight/2)));
        cv::resize(Video4Frame, Video4Shrunk, Size((VideoWidth/2), (VideoHeight/2)));

        Video1Shrunk.copyTo(CombinedVideoTop(Rect(0,0,Video1Shrunk.cols,Video1Shrunk.rows)));
        Video2Shrunk.copyTo(CombinedVideoTop(Rect(Video1Shrunk.cols,0,Video2Shrunk.cols,Video2Shrunk.rows)));
        Video3Shrunk.copyTo(CombinedVideoBottom(Rect(0,0,Video3Shrunk.cols,Video3Shrunk.rows)));
        Video4Shrunk.copyTo(CombinedVideoBottom(Rect(Video3Shrunk.cols,0,Video4Shrunk.cols,Video4Shrunk.rows)));

        CombinedVideoTop.copyTo(FullCombinedVideo(Rect(0,0,CombinedVideoTop.cols, CombinedVideoTop.rows)));
        CombinedVideoBottom.copyTo(FullCombinedVideo(Rect(0,CombinedVideoTop.rows, CombinedVideoBottom.cols,CombinedVideoBottom.rows)));


        std::string FinalFilePath = CombinedVideo.toUtf8().constData();
        cv::VideoWriter CombinedWriter(FinalFilePath, CV_FOURCC('D', 'I', 'V', 'X'),10,cvSize(FullCombinedVideo.cols, FullCombinedVideo.rows));

        CombinedWriter << FullCombinedVideo;

    }


}

The program compiles with no problem, but when I actually try to run it I get the following error:

OpenCV Error: Assertion failed (0 <= roi.x && 0 <= roi.width && roi.x + roi.width <= m.cols && 0 <= roi.y && 0 <= roi.height && roi.y + roi.height <= m.rows) in Mat, file /tmp/OpenCV-2.4.3/modules/core/src/matrix.cpp, line 322
Qt has caught an exception thrown from an event handler. Throwing
exceptions from an event handler is not supported in Qt. You must
reimplement QApplication::notify() and catch all exceptions there.

The program has unexpectedly finished.

Can anyone tell me what the problem is here? I am completely stumped. The video files I am trying to open are in .avi format, I am using QtCreator, OpenCV 2.4.3, and running it on OSX Snow Leopard.

Thanks

Are you trying to display the 4 videos inside the same window, or are you trying to create a new video file with the others inside it? — karlphillip, Feb 01 '13 at 16:37
I'm trying to create a new video file with the 4 videos inside it. — JM92, Feb 03 '13 at 22:08

score 2 · Accepted Answer · edited May 23 '17 at 12:27

Writing frames to a video file is easy and there are several examples around here.

The tricky part is assembling the 4 frames into a single one, so the following code demonstrates how to do that, and it takes into consideration that all frames can have arbitrary sizes:

// Load 4 images from the disk
cv::Mat img1 = cv::imread("test1.jpg");
cv::Mat img2 = cv::imread("test2.jpg");
cv::Mat img3 = cv::imread("test3.jpg");
cv::Mat img4 = cv::imread("test4.jpg");

// Make sure they have been loaded successfully
if (img1.empty() || img2.empty() || img3.empty() || img4.empty())
{
    std::cout << "!!! Failed to load one of the images\n";
    return -1;
}

/* Make sure they are compatible: same type, depth and # channels */

if ( (img1.type() != img2.type()) ||
     (img3.type() != img4.type()) ||
     (img1.type() != img4.type()) )
{
    std::cout << "!!! The depth doesn't match!\n";
    return -1;
}

if ( (img1.depth() != img2.depth()) ||
     (img3.depth() != img4.depth()) ||
     (img1.depth() != img4.depth()) )
{
    std::cout << "!!! The depth doesn't match!\n";
    return -1;
}

if ( (img1.channels() != img2.channels()) ||
     (img3.channels() != img4.channels()) ||
     (img1.channels() != img4.channels()) )
{
    std::cout << "!!! Number of channels doesn't match!\n";
    return -1;
}

// Create the destination image based on the size of the loaded images
//  _________
// |    |    |
// |_1__|_2__|
// |    |    |
// |_3__|_4__|

/* As the input images might have different sizes, we need to make sure
 * to create an output image that is big enough to store all of them.
 */

// Compute the width of the output image
int row1_size = img1.size().width + img2.size().width;
int row2_size = img3.size().width + img4.size().width;
int new_width = std::max(row1_size, row2_size);

// Compute the height of the output image
int col1_size = img1.size().height + img3.size().height;
int col2_size = img2.size().height + img4.size().height;
int new_height = std::max(col1_size, col2_size);

// Create the destination image
cv::Mat dst_img(cv::Size(new_width, new_height), img1.type(), cv::Scalar(0, 0, 0));
std::cout << "dst: size " << dst_img.size().width << "x" << dst_img.size().height << std::endl;

/* Copy the pixels of the input images to the destination */

//  _________
// |    |    |
// |_1__|    |
// |         |
// |_________|
img1.copyTo(dst_img(cv::Rect(0, 0, img1.cols, img1.rows))); 

//  _________
// |    |    |
// |_1__|_2__|
// |         |
// |_________|    
img2.copyTo(dst_img(cv::Rect(img1.size().width, 0, img2.cols, img2.rows)));

//  _________
// |    |    |
// |_1__|_2__|
// |    |    |
// |_3__|____|    
img3.copyTo(dst_img(cv::Rect(0, 
                             std::max(img1.size().height, img2.size().height), 
                             img3.cols, 
                             img3.rows)));

//  _________
// |    |    |
// |_1__|_2__|
// |    |    |
// |_3__|_4__|    
img4.copyTo(dst_img(cv::Rect(img3.size().width, 
                             std::max(img1.size().height, img2.size().height), 
                             img4.cols, 
                             img4.rows)));

// For testing purposes, display it on the screen
cv::imshow("Test", dst_img);
cv::waitKey(0);

These are the dimensions of the images used for testing: 64x64, 128x128, 256x256 and 320x320

Result:

enter image description here

In this demo you can notice that all images are very close to each other since my 2x2 grid doesn't have predefined dimensions. You might want to make your grid with fixed dimensions so the images stay aligned vertically & horizontally. This would be a very small change to the code. — karlphillip, Feb 04 '13 at 01:20

score 0 · Answer 2 · edited May 23 '17 at 12:04

0

Use ffmpeg (or libav) which is the proper tool for this task and will also do it faster.

See: FFMPEG 2 Videos transcoded and side by side in 1 frame?

edited May 23 '17 at 12:04

Community

1
1

answered Feb 03 '13 at 22:50

ypnos

50,202
14
95
141

score 0 · Answer 3 · answered Feb 03 '13 at 23:00

As opencv says, you have the ROI (region of interest) wrong.

Probably you have an off-by-one error so you have a region that starts at -1 or ends beyond the width/height.

Either check these int he debugger, or print the coordinates and see which one is wrong.

Combining videos in opencv and Qt

3 Answers3