3D Object selection in opengl

Question

I am currently making a 3d chess game in opengl. I still struggle with the selection of the different figures. I followed the tutorials by thinmatrix and came this far: https://i.stack.imgur.com/TRMCG.jpg.

Now I want the user to be able to select the figures by clicking on them. I have the camera position, the ray in which direction the mouse is pointing and the position of the figures. How can I detect if the ray hits the figure (probably using a rectangle hitbox) when it starts at the position of the camera?

My code so far:

public void update(Vector3f mouseRay, Camera camera, Figure figure){
    Vector3f start = camera.getPosition();
    Vector3f figurePos = figure.getPosition();
    if(intersect()){
       selectFigure();
    }
}

EDIT: I tried this: Ray-Sphere intersection but it somehow didn't work. A sphere intersection also seemed very inefficient in respect of a ray box intersection.

@Rocco that's actually a good idea but I use shaders so the colors are not consistent and they can match with the skybox / chess board txeture — luis, May 13 '21 at 14:10
What I mean with color picking is that you render the scene in a different buffer, assigning an unique solid color to each entity you want to select, removing any light effect. I'll search a link describing it since it's a long time that I don't personally code with OpenGL — Rocco, May 13 '21 at 14:14
Oh that is really smart, didn't think of that. I am pretty new to opengl so a reference would be helpful. Thanks! — luis, May 13 '21 at 14:16
http://www.opengl-tutorial.org/miscellaneous/clicking-on-objects/picking-with-an-opengl-hack/ unfortunately it's not Java, but I think that the concept is clear, a custom fragment shader must produce an uniform color — Rocco, May 13 '21 at 14:21
see [OpenGL 3D-raypicking with high poly meshes](https://stackoverflow.com/a/51764105/2521214) ... the 8bit stencil is ideal for you as you got just 32 objects ... no need to any heavy change of code just add the stencil commands setting which piece is rendering ... — Spektre, May 14 '21 at 07:20

score 0 · Answer 1 · answered May 27 '22 at 21:21

You'll have to follow following steps (I'm assuming you are aware of rendering pipeline and aware of OpenGL/WebGL)

Get the list of all the objects you have.
Assign every object a unique color. Following would be an easy way to assign the unique color based on index of the object in the list.

    int i = // Index of the object  
    // We've added 1 to the index because 0 index is translated to black color 
    // And our background is also rendered as black so we skip will that color.
    int r = (i + 1 & 0x000000FF) >> 0;
    int g = (i + 1 & 0x0000FF00) >> 8;
    int b = (i + 1 & 0x00FF0000) >> 16;
    glm::vec4 unique_color = glm::vec4(r / 255.0f, g / 255.0f, b / 255.0f, 1.0);

Create a frame-buffer and render all the objects with their uniquely assigned solid colors.
When the rendering is complete, you now read the click position pixel color from the rendered frame buffer texture.
Decode the color into index of object back like given below. (This is exactly revers of what we've done in step 2)

    int triangle_index =
        color.r +
        color.g * 256 +
        color.b * 256 * 256;

With this index you have the selected object from the initial list of all objects.
You can read more about this technique here, http://www.opengl-tutorial.org/miscellaneous/clicking-on-objects/picking-with-an-opengl-hack/

3D Object selection in opengl

1 Answers1