1

I am trying to open a yuv format image. I can open it with rawpixels.net and display it after setting the following

width:1920
height:1080
predefined format: yuv420 (nv12)
pixel format yuv

But if I open with opencv with the following code I failed to open.

#include <iostream>
#include <opencv2/core.hpp>
#include <opencv2/opencv.hpp>

int main() {
    std::cout << "OpenCV version: " << CV_VERSION << std::endl;


    cv::Mat image = cv::imread("camera_capture_256_2020_10_07_11_11_02.yuv");
    if (image.empty() == true) {

        std::cout << "image empty"<< std::endl;

        return 0;
    }   
        
    cv::imshow("opencv_logo", image);
    cv::waitKey(0);    

    return 0;
}

The program prints as "image empty".

I am puzzled why I can't open the file with opencv.

The sample image is found here.

The yuv image opened with rawpixels.net would look like this.

enter image description here

Thanks,

takanoha
  • 183
  • 4
  • 17
  • 1
    yes. here [link](https://drive.google.com/file/d/1H__wD_ygidxNCUAJ4S5LvzE7clqnhmX9/view?usp=sharing) – takanoha Jan 15 '21 at 07:19
  • I looked at the linked file `camera_preview_250_2020_10_07_11_11_02.yuv` (which isn't the same you're trying to open in your example) in YUView 2.9 and it looks really blurry. Is that's how it's supposed to be? It makes it hard to visually confirm that one has successfully read the image. Also: In your example code it looks like you're trying to convert FROM `jpg` TO `yuv` so the title ("_opencv can't open a yuv422 image_") is a bit confusing. – Ted Lyngmo Jan 15 '21 at 08:34
  • 1
    I'm not aware that **OpenCV** is able to write YUV files like that - there is no mention of it in the documentation for `cv2.imwrite()`. I think you need to do `YUV = cvtColor(...BGR2YUV...)` and use standard Python `write(YUV.tobytes())` – Mark Setchell Jan 15 '21 at 08:35
  • http://rawpixels.net/ makes the image look green if yuv422 (which you mention in the title) is used, but sort of ok if yuv420 is used (that you mention in the body of the question). – Ted Lyngmo Jan 15 '21 at 08:43
  • 1
    If your YUV image is 1920x1080 and `nv12` format, it should be 1920*1080*1.5 bytes because `nv12` is 12 bits/pixel. That makes 3110400 bytes, but your file is 8355840 bytes, or 2.7x too big? – Mark Setchell Jan 15 '21 at 14:08
  • @TedLyngmo Thanks for your message. I just updated the note. I The main problem here is opening a yuv file. I made a mistake on the description at first (eg. the code I pasted was opening .jpg file but what I intended to do was to open .yuv file and also I don't need to call convert() ). I also added a screenshot of the image opened with rawpixels.net. It is a bit blurry as you mentioned but this is the picture. – takanoha Jan 16 '21 at 00:25
  • @MarkSetchell Thanks for mentioning the proper size of the yuv, but when calling imcread() I don't need to specify the image size as the api takes care of it. I wonder if it is actually possible to load the yuv image with imread() – takanoha Jan 16 '21 at 00:27
  • @takanoha Opening the file isn't a problem. I read it fine. As Mark Stechell mentioned, there's some scaling and you need to convert it - but reading it wasn't a problem. You can't do it directly with OpenCV though (afaik), but there are conversion constants for a lot of YUV variants in there. – Ted Lyngmo Jan 16 '21 at 00:27
  • @TedLyngmo Hmm really ? When I tried to open the file the Mat object is empty. I complied just fine with following script : g++ hello.cpp -o hello -I/usr/local/include/opencv4 -L/usr/local/opencv/build_linux/lib \ -lopencv_core \ -lopencv_highgui \ -lopencv_imgproc \ -lopencv_imgcodecs I wonder what the problem is... – takanoha Jan 16 '21 at 00:31
  • Yeah, OpenCV can't interpret it directly, that much is clear (from what I've read) - but reading it raw isn't a problem. One just has to read it into memory and feed it to OpenCV:s conversion function with the correct parameters. I didn't get that far Today though, but I found a few similar problems here at SO with hints. The 1.5 Mark mentioned is one of them. – Ted Lyngmo Jan 16 '21 at 00:37
  • @TedLyngmo I see. Could you please elaborate how you opened the file ? – takanoha Jan 16 '21 at 00:41
  • @takanoha Sure, but it doesn't have anything to do with OpenCV. It's just classic reading the file into memory. [example](https://godbolt.org/z/vPMWr5) – Ted Lyngmo Jan 16 '21 at 00:42
  • @MarkSetchell - I'm looking in the file itself. It's funny you should talk about a mis-match in the size - there's 3,125,760 bytes at the start of the file that appear to hold something useful, with the remainder of the file all being zeros. – enhzflep Jan 16 '21 at 00:45
  • Furthermore, 1920x1080 = 2,073,600 Multiply this by 4 (bytes per pixel) and we still only get 8,294,400 bytes, yet the downloaded file is some 8,355,840 bytes – enhzflep Jan 16 '21 at 00:49
  • 1
    @enhzflep Interesting. In fact when I opened with rawpixels.net a green line is seen that is probably the mysterious header. – takanoha Jan 16 '21 at 00:50
  • 1
    One trick when doing this kind of development is to start with super clear data. A square box perfectly matching the dimensions, colors that you can verify etc. When I did a similar thing to extract data from damaged CDROM:s I burned a pattern over the whole disk and took a drilling machine and made a big hole in it. – Ted Lyngmo Jan 16 '21 at 00:53
  • 1
    @takanoha - Then the question seems somewhat unhelpful, since it appears to boil down to the following: "I have a broken YUV file that I can open with RawPixels.Net and I want to use it with openCV. Why does OpenCV refuse to work with this image?" - Ted's comment about having known-good data is the starting-point. I suggest you return to there and don't start trying again until you've got good data. I certainly wouldn't have dipped my toe in if I'd realized there's a problem right from the word GO. – enhzflep Jan 16 '21 at 00:57

1 Answers1

3

The very first thing to do when dealing with raw (RGB, BGR, YUV, NV12 and others) images is to know the dimensions in pixels of the image - you are really quite lost without those - though you can do certain tricks to look for correlation to find the row width since each row is essentially similar to the one above normally.

The next thing is to check the filesize is correct. So if it is RGB and 8-bit 1920x1080, your file must be 1920x1080x3 pixels in size - if not there is a problem. Your image is 1920x1080 and NV12 which is 12-bits or 1.5 bytes per pixel, so I expect your file to be 1920x1080*1.5 bytes. It is not that, so there is immediately a problem. There is either a header, or multiple frames or trailing data or some other issue.

So, where is the image data in the file? At the start? At the end? One way to solve this is to look at the file as though it was purely a greyscale image and see if there are large blocks of black which are zero bytes or padding. As there is no known image size, I generally take the file size in bytes and go to Wolfram Alpha website and type in "factors of XXX" where XXX is the file size and then choose 2 numbers near the square-root of the file size so I get a square-ish image. So for yours, I chose 2720x3072 and treated your file as a single greyscale image of that size. Using ImageMagick in Terminal:

magick -depth 8 -size 2720x3072 gray:camera_preview_250_2020_10_07_11_11_02.yuv image.jpg

enter image description here

I can see, at a glance that the data are at the start of the file and the end of the file is zero-padding, i.e. black. If the black had been at the start of the image, I would have taken the final H x W x 1.5 bytes.

Another alternative for this step, is to take the file size in bytes and divide it by the image width to get a number of lines and see how that looks. So your file is 8355840 bytes, that would be 8355840/1920 or 4,325 lines. Let's try that:

magick -depth 8 -size 1920x4352 gray:camera_preview_250_2020_10_07_11_11_02.yuv image.jpg

enter image description here

That is very encouraging because we can see the Y (greyscale) image at the start of the file and some lower-resolution UV channels following, and the fact that there are not 2 separate channels following probably means they are interlaced, alternating U and V samples rather than planar U samples followed by V samples.

Ok, if your data is YUV or NV12, the best tool for that is ffmpeg. We already know that the data is at the start of the file and we know the dimensions and the format. We also know that there is padding after the image, so we need to just take the first frame like this:

ffmpeg -s 1920x1080 -pix_fmt nv12 -i cam*yuv -frames:v 1 image.png

enter image description here

Now we have confidence about the dimensions and format, we need OpenCV to read that. The normal cv2.imread() cannot read that because it is just raw data, and unlike JPEG or PNG or TIFF, there is no image height and width in a header - it is just pure sensor data.

So, you need to use the regular C/C++ read() system call to get the first 1920x1080x1.5 bytes. Then you need to call cv2.cvtColor() on the received buffer to convert it to a regular BGR format Mat.

Mark Setchell
  • 191,897
  • 31
  • 273
  • 432