OpenCV get / set IplImage RGB values

Question

To begin, just note that I'm avare of Accessing certain pixel RGB value in openCV and Pixel access in OpenCV 2.2. Neither of those address my problem.

What I would like to do, is to reliably read / write the pixel RGB values in OpenCV's IplImage. So far, I have seen this:

uint8_t cr = CV_IMAGE_ELEM(color_img, uint8_t, y, x * color_img->nChannels + 2);
uint8_t cg = CV_IMAGE_ELEM(color_img, uint8_t, y, x * color_img->nChannels + 1);
uint8_t cb = CV_IMAGE_ELEM(color_img, uint8_t, y, x * color_img->nChannels + 0);

Now that seems plain wrong to me.

const uint8_t *p_color_pixel = cvPtr2D(color_img, y, x, NULL);
uint8_t cr = p_color_pixel[2], cg = p_color_pixel[1], cb = p_color_pixel[0];

This is still a bit painful, but probably the fastest way. One could also use cvGet2D(), although that also seems to involve the awkward multiplication of x by the number of channels.

Anyway, my issue is that those are all quite "unsafe", in the sense that the image may be RGB or BGR or ARGB. Also, the pixels may not be uint8_t. So the result feels still pretty random, depending on what kind of image and format I'm reading. I'm actually processing images that store some 24-bit integer values, packed to RGB, so the correct order is crucial (a wrong order will essentially yield noise). The only approach I've seen that seems "semi-safe" so far is quite morbid, both code and performance-wise:

CvSize t_image_size = cvGetSize(image);
IplImage *ir = cvCreateImage(t_image_size, 8, 1);
IplImage *ig = cvCreateImage(t_image_size, 8, 1);
IplImage *ib = cvCreateImage(t_image_size, 8, 1);
cvSplit(image, left_b, left_g, left_r, 0);

double cr = cvGetReal2D(ir, x, y),
    cg = cvGetReal2D(ig, x, y), cvGetReal2D(ib, x, y);

cvReleaseImage(&ir);
cvReleaseImage(&ig);
cvReleaseImage(&ib);

I mean ... come on, we are in the 21st century, why is it so damn hard to get correct RGB?!

Note that I'd prefer using the "C" interface, as the C++ one is quite unstable at the moment. But if there was a reliable way in C++, I'd take it.

Answer 1

So anyway, here's what we know. From IplImage documentation, we can see a few relevant fields:

nChannels // Number of channels. Most OpenCV functions support 1-4 channels.
depth // The supported depths are: IPL_DEPTH_8U, IPL_DEPTH_16U, ...
channelSeq // Ignored by OpenCV
colorModel // Ignored by OpenCV
dataOrder // 0 - interleaved color channels, 1 - separate color channels.
          // CreateImage() only creates images with interleaved channels.
          // For example, the usual layout of a color image is: b00r00g00b10r10g10

So at least the depth and the usual layout is knowable. What about the unusual layouts, though? We can also check imread documentation. Quite at the end, it says:

Note: In the case of color images, the decoded images will have the channels stored in B G R order.

So that seems to be it. There seems to be no "unusual" layout implemented in reading the images.

Let's see how cvGet2D() is implemented (OpenCV 2.4.7):

CV_IMPL CvScalar cvGet2D(const CvArr* arr, int y, int x)
{
    CvScalar scalar = {{0,0,0,0}};
    int type = 0;
    uchar* ptr;

    if(CV_IS_MAT(arr)) {
        CvMat* mat = (CvMat*)arr;
        if((unsigned)y >= (unsigned)(mat->rows) ||
           (unsigned)x >= (unsigned)(mat->cols))
            CV_Error(CV_StsOutOfRange, "index is out of range");
        type = CV_MAT_TYPE(mat->type);
        ptr = mat->data.ptr + (size_t)y * mat->step + x * CV_ELEM_SIZE(type);
    } else if(!CV_IS_SPARSE_MAT(arr))
        ptr = cvPtr2D(arr, y, x, &type);
    else {
        int idx[] = {y, x};
        ptr = icvGetNodePtr((CvSparseMat*)arr, idx, &type, 0, 0);
    }
    if(ptr)
        cvRawDataToScalar(ptr, type, &scalar);
    return scalar;
}

That's actually interesting. Debugging reveals that on IplImage, this calls cvPtr2D and then cvRawDataToScalar. Further debugging also reveals that channelSeq of IplImage is actually a string (damn you, OpenCV implementation, why did you keep that a secret?!). So a robust code for reading OpenCV image RGB values is:

const char *rpos = strchr(color_img->channelSeq, "R"),
           *gpos = strchr(color_img->channelSeq, "G"),
           *bpos = strchr(color_img->channelSeq, "B");
if(!rpos || !gpos || !bpos)
    throw std::runtime_error("unknown image channel sequence");
const int rIdx = rpos - color_img->channelSeq,
          gIdx = gpos - color_img->channelSeq,
          bIdx = bpos - color_img->channelSeq;
// determine channel mapping

CvScalar color = cvGet2D(color_img, y, x); // note y, x
uint8_t cr = uint8_t(color.val[rIdx]),
        cg = uint8_t(color.val[gIdx]),
        cb = uint8_t(color.val[bIdx]);
// do not multiply by 255, it is already in that range

This will correctly handle any channel order and any depth. Or if you want to go faster, use imread(filename, CV_LOAD_IMAGE_COLOR) to convert the values to 8 bit per pixel. Then this can be used:

assert(color_img->depth == IPL_DEPTH_8U); // note that imread does not really say whether it converts always to IPL_DEPTH_8U or sometimes to IPL_DEPTH_8S!
const uint8_t *p_color_pixel = cvPtr2D(color_img, y, x, NULL);
uint8_t cr = p_color_pixel[rIdx], cg = p_color_pixel[gIdx],
    cb = p_color_pixel[bIdx];

And that's it. It's not pretty, but I guess that comes with using OpenCV. Note that it might actually not be reliable as the documentation says that OpenCV is not using the IplImage::channelSeq field, but this is the best one can do now. At this point I don't trust anything that's written in OpenCV documentation, it's the worst.

I was kind of hoping that there will be a way of doing this, say cvGetRGBA2D() that would return CvScalar with RGBA order always (one line instead of my 12 lines). Ridiculous.