1

I would like to grab an OpenGL image and feed it to OpenCV for analysis (as a simulator for the OpenCV algorithms) but I am not finding much information about it, all I can find is the other way around (placing an OpenCV image inside OpenGL). Could someone explain how to do so?

EDIT:

I will be simulating a camera on top of a Robot, so I will render in realtime a 3D environment and display it in a Qt GUI for the user. I will give the user the option to use a a real webcam feed or a simulated 3D scene (that changes as the robot moves) and the OpenCV algorithm will be the same for both inputs so the user might test his code without having to use a real robot all the time.

Michel Feinstein
  • 13,416
  • 16
  • 91
  • 173
  • 1
    What does "OpenGL image" mean in this context? A texture? A scene rendered to the framebuffer? A scene rendered to some off-screen buffer? – Mats Petersson Oct 23 '13 at 23:28
  • @MatsPetersson I updated the question with more explanations – Michel Feinstein Oct 24 '13 at 05:10
  • @MatsPetersson I guess its the "a scene rendered to the framebuffer" option, to be more clear. I am still learning OpenGL more deeply so I was trying to find more information about all the stuff I will have to read, and I couldn't find much...any orientatios is most welcomed! – Michel Feinstein Oct 24 '13 at 05:34
  • Right, so if you render to an off-screen buffer, you should be able to convert that off-screen buffer to an image (in fact, as long as you know enough about your off-screen buffer, it should be possible to create an IPLImage from that, I think). – Mats Petersson Oct 24 '13 at 07:15
  • @MatsPetersson if I render to an off-screen buffer I wont be seeing the scene being rendered right? Why can't I do it with an on-screen buffer? – Michel Feinstein Oct 24 '13 at 13:09
  • Because the onscreen buffer isn't trivially useable. You can quite easily use this offscreen buffer as a texture to draw to the screen as well. – Mats Petersson Oct 25 '13 at 06:33

3 Answers3

3

You are probably looking for the function glReadPixels. It will download whatever is currently displayed by OpenGL to a buffer.

unsigned char* buffer = new unsigned char[width*height*3];
glReadPixels(0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE, buffer);
cv::Mat image(height, width, CV_8UC3, buffer);
cv::imshow("Show Image", image);

For OpenCV you will probably also need to flip and convert to BGR as well.

Edit: Since just using glReadPixels is not a very efficient way to do it, here is some sample code using Framebuffers and Pixel Buffer Objects to efficiently transfer: How to render offscreen on OpenGL?

Community
  • 1
  • 1
littleimp
  • 1,159
  • 9
  • 13
  • I am new to OpenCV, is BGR something specific or just RGB backwards? – Michel Feinstein Oct 24 '13 at 13:07
  • 2
    Just beware that glReadPixels is an excellent way to make your $300 graphics card to produce 5 fps, because every time you call glReadPixels, the graphics chip will have to drain the whole pipeline of work. If you've heard of "branch mispredicts" in CPU's, this is the same sort of thing, but 10000x worse. – Mats Petersson Oct 25 '13 at 06:32
  • @MatsPetersson damn, it looked like a clean solution...do you have any bettert options, maybe some code that I can search about? – Michel Feinstein Oct 25 '13 at 06:59
  • Like I said, draw off-screen, then just make a quad that you texture with the off-screen drawing. – Mats Petersson Oct 25 '13 at 07:05
  • Of course, if you don't need 100+ fps, and you are not drawing 20 million triangles per frame, it may well work just fine. I'm just pointing out it's a bit like driving a Ferrari sports-car in heavy city traffic - you don't get to use the full power very often, or for very long... – Mats Petersson Oct 25 '13 at 07:07
  • Yeah BGR is just a different order of the channels. You will quickly notice that opencv uses BGR when using imshow to display an image. – littleimp Oct 25 '13 at 11:13
  • I agree that glReadPixels is a naive implementation. Here is the right way to do it: http://stackoverflow.com/questions/12157646/how-to-render-offscreen-on-opengl – littleimp Oct 25 '13 at 11:18
2

I did it in a previous research project. There are not much difficulties here.

What you have to do is basically:

  • make a texture read from OpenGL to some pre-allocated memory buffer;
  • apply some geometric transform (flip X and/or Y coordinate) to account for the possibly different coordinate frames between OpenGL and OpenCV. It's a detail but it helps in visualization (hint: use a texture with an F letter inside to find quickly what coordinate you need to flip!);
  • you can build an OpenCV cv::Matobject directly around your pre-allocated memory buffe, and then process it directly or copy it to some other matrix object and process it.

As indicated in another answer, reading OpenGL texture is a simple matter of calling the glRead() function. What you get is usually 3 or 4 channels with 8 bits per data (RGB / RGBA - 8 bits per channel), though it may depend on your actual OpenGL context.

If color is important to you, you may need (but it is not required) to convert the RGB image data to the BGR format (Blue - Green - Red). For historical reasons, this is the default color channel ordering in OpenCV. You do this with a call to cv::cvtColor(source, dest, cv::COLOR_RGB2BGR) for example.

sansuiso
  • 9,259
  • 1
  • 40
  • 58
2

I needed this for my research. I took littleimp's advice, but fixing the colors and flipping the image took valuable time to figure out.

Here is what I ended up with.

typedef Mat Image ;

typedef struct {
  int width;
  int height;
  char* title;
  float field_of_view_angle;
  float z_near;
  float z_far;
} glutWindow;

glutWindow win;

Image glutTakeCVImage() {
  // take a picture within glut and return formatted
  // for use in openCV
  int width = win.width;
  int height = win.height;
  unsigned char* buffer = new unsigned char[width*height*3];
  glReadPixels(0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE, buffer);
  Image img(height, width, CV_8UC3, buffer);
  Image flipped_img;
  flip(img,flipped_img,0);
  Image BGR_img;
  cvtColor(flipped_img,BGR_img, COLOR_RGB2BGR);
  return BGR_img;
}

I hope someone finds this useful.

protist
  • 1,172
  • 7
  • 9