glGetPixels on Offscreen framebuffer opengl

Question

I generate a PointCloud in my program, and now, I want to be able to click on a point in this point cloud rendered to my screen using OpenGL.

In order to do so, I used the trick of giving to each pixel in an offscreen render a colour based on its index in the VBO. I use the same camera for my offscreen render and my onscreen render so they move together, and when I click, I get values of my offscreen render to retrieve the position in the VBO to get the point I clicked on. This is the theory since when I click, I have only (0,0,0). I believe that means my FBO is not well renderer but I'm not sure whether it is that or if the problem comes from somewhere else...

So here are the steps. clicFBO is the FBO I'm using for offscreen render, and clicTextureColorBuf is the texture in which I write in the FBO

glGenFramebuffers(1, &clicFBO);
glBindFramebuffer(GL_FRAMEBUFFER, clicFBO);
glGenTextures(1, &clicTextureColorBuf);
glBindTexture(GL_TEXTURE_2D, clicTextureColorBuf);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, SCR_WIDTH, SCR_HEIGHT, 0, GL_RGB, GL_UNSIGNED_BYTE, NULL);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, clicTextureColorBuf, 0);
GLenum DrawBuffers[1] = { GL_COLOR_ATTACHMENT0 };
glDrawBuffers(1, DrawBuffers);

After that, I wrote a shader that gives to each point the color of its index in the VBO...

std::vector<cv::Point3f> reconstruction3D; //Will contain the position of my points
std::vector<float> indicesPointsVBO; //Will contain the indexes of each point
for (int i = 0; i < pts3d.size(); ++i) {
    reconstruction3D.push_back(pts3d[i].pt3d);
    colors3D.push_back(pt_tmp);
    indicesPointsVBO.push_back(((float)i / (float)pts3d.size() ));
}

GLuint clicVAO, clicVBO[2];
glGenVertexArrays(1, &clicVAO);
glGenBuffers(2, &clicVBO[0]);
glBindVertexArray(clicVAO);
glBindBuffer(GL_ARRAY_BUFFER, clicVBO[0]);
glBufferData(GL_ARRAY_BUFFER, reconstruction3D.size() * sizeof(cv::Point3f), &reconstruction3D[0], GL_STATIC_DRAW);
glEnableVertexAttribArray(0);
glVertexAttribPointer(0, 3, GL_FLOAT, GL_FALSE, 0, (GLvoid*)0);
glEnable(GL_PROGRAM_POINT_SIZE);

glBindBuffer(GL_ARRAY_BUFFER, clicVBO[1]);
glBufferData(GL_ARRAY_BUFFER, indicesPointsVBO.size() * sizeof(float), &indicesPointsVBO[0], GL_STATIC_DRAW);
glEnableVertexAttribArray(1);
glVertexAttribPointer(1, 1, GL_FLOAT, GL_FALSE, 0, (GLvoid*)0);
glBindBuffer(GL_ARRAY_BUFFER, 0);
glBindVertexArray(0);
glBindFramebuffer(GL_FRAMEBUFFER, 0);

and the vertex shader:

layout (location = 0) in vec3 pos;
layout (location = 1) in float col;

out float Col;
uniform mat4 model;
uniform mat4 view;
uniform mat4 projection;
uniform int pointSize;

void main()
{
    gl_PointSize = pointSize;
    gl_Position =  projection * view * model * vec4(pos, 1.0);

    Col = col;
}

And the Fragment:

#version 330 core
layout(location = 0) out vec4 FragColor;
in float Col;
void main()
{
    FragColor = vec4(Col, Col, Col ,1.0);
}

And this is how I render this texture:

    glm::mat4 view = camera.GetViewMatrix();
    glm::mat4 projection = glm::perspective(glm::radians(camera.Zoom), (float)SCR_WIDTH / (float)SCR_HEIGHT, 1.0f, 100.0f);
    glBindFramebuffer(GL_FRAMEBUFFER, clicFBO);
    clicShader.use();

    glDisable(GL_DEPTH_TEST);
    glClearColor(1.0f, 1.0f, 1.0f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT);
    clicShader.setMat4("projection", projection);
    clicShader.setMat4("view", view);
     model = glm::mat4();
    clicShader.setMat4("model", model);
    clicShader.setInt("pointSize", pointSize);

    glBindVertexArray(clicVAO);
    glDrawArrays(GL_POINTS, 0, (GLsizei)reconstruction3D.size());
    glBindFramebuffer(GL_FRAMEBUFFER, 0);

And then, when I click, I Use this piece of Code:

glBindFramebuffer(GL_FRAMEBUFFER, clicFBO);
glReadBuffer(GL_COLOR_ATTACHMENT0);
int width = 11, height = 11;
std::array<GLfloat, 363> arry{ 1 };

glReadPixels(Xpos - 5, Ypos - 5, width, height, GL_RGB, GL_UNSIGNED_BYTE, &arry);
for (int i = 0; i < 363; i+=3) { // It's 3 time the same number anyways for each number
    std::cout << arry[i] << " "; // It gives me only 0's
}
std::cout << std::endl << std::endl;
glBindFramebuffer(GL_FRAMEBUFFER, clicFBO);

I know the error might be really stupid but I still have some problems with how OpenGL works.

I put what I thought was necessary to understand the problem (without extending too much), but if you need more code, I can write it too.

I know this is not a question in which you can say Yes or No and it's more like debugging my program, but since I really don't find from where the problem comes from, I'm looking toward someone who can explain to me what I did wrong. I do not necessarily seek the solution itself, but clues that could help me understand where my error is ...

Answer 1

Using a framebuffer object FBO to store a "object identifier" is a cool method. But also want to see the objects, right? Then you must render also to the default frame buffer (let me call it "defFB", which is not a FBO).

Because you need to render to two different targets, you need one of these techniques:

• Draw objects twice (e.g. with two glDrawArrays calls), one to the FBO and a second one to the defFB.
• Draw to two FBO's images at once and later blit one of then (with colors) to the defFB.

For the first technique you may use a texture attached to a FBO (as you currently do). Or you can use a "Renderbuffer" and draw to it.

The second approach needs a second "out" in the fragment shader:

layout(location = 0) out vec3 color; //GL_COLOR_ATTACHMENT0
layout(location = 1) out vec3 objID; //GL_COLOR_ATTACHMENT1

and setting the two attachments with glDrawBuffers.

For the blit part, read this answer.

Note that both "out" have the same format, vec3 in this example.

A fail in your code is that you set a RGB texture format and also use this format at glReadPixels, but your "out" in the FS is vec4 instead of vec3.

More concerns are:

• Check the completeness with glCheckFramebufferStatus
• Using a "depth attachment" to the FBO may be needed, even it will not be used for reading.
• Disabling the depth test will put all elements if the frame. Your point-picking will select the last drawn, not the nearest.

Answer 2

I found the problem.
There were 2 failures in my code :
The first one is that in OpenGL, there is an Y inversion between the image and the framebuffer. So in order to pick the good point, you have to flip Y using the size of the viewport : I did it like this :

GLint m_viewport[4];
glGetIntegerv(GL_VIEWPORT, m_viewport);
int YposTMP = m_viewport[3] - Ypos - 1;

The second one is the use of glReadPixels(Xpos - 2, Ypos - 2, width, height, GL_RGB, GL_UNSIGNED_BYTE, &pixels[0]);, the 6th parameter must be GL_FLOAT since the datas i'm returning are float.

Thanks all!
Best regards,
R.S