cv::imshow in opencv is only displaying parts of a composite image, but displaying the parts separately works. Why?

Question

Objective and problem

I'm trying to process a video file on the fly using OpenCV 3.4.1 by grabbing each frame, converting to grayscale, then doing Canny edge detection on it. In order to display the images (on the fly as well), I created a Mat class with 3 additional headers that is three times as wide as the original frame. The 3 extra headers represent the images I would like to display in the composite, and are positioned to the 1st, 2nd and 3rd horizontal segment of the composite.

After image processing however, the display of the composite image is not as expected: the first segment (where the original frame should be) is completely black, while the other segments (of processed images) are displayed fine. If, on the other hand, I display the ROIs one by one in separate windows, all the images look fine.

These are the things I tried to overcome this issue:

1. use .copyTo to actually copy the data into the appropriate image segments. The result was the same.
2. I put the Canny image to the compOrigPart ROI, and it did display in the first segment, so it is not a problem with the definition of the ROIs.
3. • Define the composite as three channel image
   • In the loop convert it to grayscale
   • put processed images into it
   • convert back to BGR
   • put the original in.

This time around the whole composite was black, nothing showed.

4. As per gameon67's suggestion, I tried to create a namedWindow as well, but that doesn't help either.

Code:

int main() {

    cv::VideoCapture vid("./Vid.avi");
    if (!vid.isOpened()) return -1;

    int frameWidth = vid.get(cv::CAP_PROP_FRAME_WIDTH);
    int frameHeight = vid.get(cv::CAP_PROP_FRAME_HEIGHT);
    int frameFormat = vid.get(cv::CAP_PROP_FORMAT);

    cv::Scalar fontColor(250, 250, 250);
    cv::Point textPos(20, 20);

    cv::Mat frame;

    cv::Mat compositeFrame(frameHeight, frameWidth*3, frameFormat);
    cv::Mat compOrigPart(compositeFrame, cv::Range(0, frameHeight), cv::Range(0, frameWidth));
    cv::Mat compBwPart(compositeFrame, cv::Range(0, frameHeight), cv::Range(frameWidth, frameWidth*2));
    cv::Mat compEdgePart(compositeFrame, cv::Range(0, frameHeight), cv::Range(frameWidth*2, frameWidth*3));


    while (vid.read(frame)) {
        if (frame.empty()) break;

        cv::cvtColor(frame, compBwPart, cv::COLOR_BGR2GRAY);
        cv::Canny(compBwPart, compEdgePart, 100, 150);
        compOrigPart = frame;

        cv::putText(compOrigPart, "Original", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
        cv::putText(compBwPart, "GrayScale", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
        cv::putText(compEdgePart, "Canny edge detection", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);

        cv::imshow("Composite of Original, BW and Canny frames", compositeFrame);
        cv::imshow("Original", compOrigPart);
        cv::imshow("BW", compBwPart);
        cv::imshow("Canny", compEdgePart);
        cv::waitKey(33);
    }
}

Questions

• Why can't I display the entirety of the composite image in a single window, while displaying them separately is OK?
• What is the difference between these displays? The data is obviously there, as evidenced by the separate windows.
• Why only the original frame is misbehaving?

Answer 1

Your compBwPart and compEdgePart are grayscale images so the Mat type is CV8UC1 - single channel and therefore your compositeFrame is in grayscale too. If you want to combine these two images with a color image you have to convert it to BGR first and then fill the compOrigPart.

while (vid.read(frame)) {
  if (frame.empty()) break;

  cv::cvtColor(frame, compBwPart, cv::COLOR_BGR2GRAY);
  cv::Canny(compBwPart, compEdgePart, 100, 150);
  cv::cvtColor(compositeFrame, compositeFrame, cv::COLOR_GRAY2BGR);
  frame.copyTo(compositeFrame(cv::Rect(0, 0, frameWidth, frameHeight)));

  cv::putText(compOrigPart, "Original", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor); //the rest  of your code

Answer 2

This is a combination of several issues.

The first problem is that you set the type of compositeFrame to the value returned by vid.get(cv::CAP_PROP_FORMAT). Unfortunately that property doesn't seem entirely reliable -- I've just had it return 0 (meaning CV_8UC1) after opening a color video, and then getting 3 channel (CV_8UC3) frames. Since you want to have the compositeFrame the same type as the input frame, this won't work.

To work around it, instead of using those properties, I'd lazy initialize compositeFrame and the 3 ROIs after receiving the first frame (based on it's dimensions and type).

---

The next set of problems lies in those two statements:

cv::cvtColor(frame, compBwPart, cv::COLOR_BGR2GRAY);
cv::Canny(compBwPart, compEdgePart, 100, 150);

In this case assumption is made that frame is BGR (since you're trying to convert), meaning compositeFrame and its ROIs are also BGR. Unfortunately, in both cases you're writing a grayscale image into the ROI. This will cause a reallocation, and the target Mat will cease to be a ROI.

To correct this, use temporary Mats for the grayscale data, and use cvtColor to turn it back to BGR to write into the ROIs.

---

Similar problem lies in the following statement:

compOrigPart = frame;

That's a shallow copy, meaning it will just make compOrigPart another reference to frame (and therefore it will cease to be a ROI of compositeFrame).

What you need is a deep copy, using copyTo (note that the data types still need to match, but that was fixed earlier).

---

Finally, even though you try to be flexible regarding the type of the input video (judging by the vid.get(cv::CAP_PROP_FORMAT)), the rest of the code really assumes that the input is 3 channel, and will break if it isn't.

At the least, there should be some assertion to cover this expectation.

---

Putting this all together:

#include <opencv2/opencv.hpp>

int main()
{
    cv::VideoCapture vid("./Vid.avi");
    if (!vid.isOpened()) return -1;

    cv::Scalar fontColor(250, 250, 250);
    cv::Point textPos(20, 20);

    cv::Mat frame, frame_gray, edges_gray;
    cv::Mat compositeFrame;
    cv::Mat compOrigPart, compBwPart, compEdgePart; // ROIs

    while (vid.read(frame)) {
        if (frame.empty()) break;

        if (compositeFrame.empty()) {
            // The rest of code assumes video to be BGR (i.e. 3 channel)
            CV_Assert(frame.type() == CV_8UC3);
            // Lazy initialize once we have the first frame
            compositeFrame = cv::Mat(frame.rows, frame.cols * 3, frame.type());
            compOrigPart = compositeFrame(cv::Range::all(), cv::Range(0, frame.cols));
            compBwPart = compositeFrame(cv::Range::all(), cv::Range(frame.cols, frame.cols * 2));
            compEdgePart = compositeFrame(cv::Range::all(), cv::Range(frame.cols * 2, frame.cols * 3));
        }

        cv::cvtColor(frame, frame_gray, cv::COLOR_BGR2GRAY);
        cv::Canny(frame_gray, edges_gray, 100, 150);

        // Deep copy data to the ROI
        frame.copyTo(compOrigPart);
        // The ROI is BGR, so we need to convert back
        cv::cvtColor(frame_gray, compBwPart, cv::COLOR_GRAY2BGR);
        cv::cvtColor(edges_gray, compEdgePart, cv::COLOR_GRAY2BGR);

        cv::putText(compOrigPart, "Original", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
        cv::putText(compBwPart, "GrayScale", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);
        cv::putText(compEdgePart, "Canny edge detection", textPos, cv::FONT_HERSHEY_PLAIN, 1, fontColor);

        cv::imshow("Composite of Original, BW and Canny frames", compositeFrame);
        cv::imshow("Original", compOrigPart);
        cv::imshow("BW", compBwPart);
        cv::imshow("Canny", compEdgePart);
        cv::waitKey(33);
    }
}

Screenshot of the composite window (using some random test video off the web):