Issue with Picking (custom unProject() function)

Question

I'm currently working on a STL file viewer. This one use an Arcball camera :

To provide more features on this viewer (which can handle more than one object) I would like to implement a click select. To achieve it, I have used picking(Pseudo code I have used)

At this time, my code to check for a any object 3D between 2 points works. However the conversion of mouse position to a correct set of vector is far away from working:

glm::vec3 range = transform.GetPosition() + ( transform.GetFront() * 1000.0f);
// x and y are cursor position on the screen             
glm::vec3 start = UnProject(x,y, transform.GetPosition().z);
glm::vec3 end = UnProject(x,y,range.z);
            
/*
    The code which iterate over all objects in the scene and checks for collision
    between my start / end and the object hitbox
*/

As you can see I have tried (maybe it is stupid) to set a the z distance between my start and my end to 100 * theFront vector of my camera. But it's not working the set of vectors I get are incoherents.

By example, placing the camera at 0 0 0 with a front of 0 0 -1 give me this set of Vectors :

Start : 0.0000~ , 0.0000~ , 0.0000~

End : 0.0000~ , 0.0000~ , 0.0000~

which is (by my logic) incoherent, I would have expected something more like (Start : 0, 0, 0) ( End : 0, 0, -1000)

I think there's an issue with my UnProject function :

glm::vec3 UnProject(float winX, float winY, float winZ)
    {
        // Compute (projection x modelView) ^ -1:
        glm::mat4 modelView = GetViewMatrix() * glm::mat4(1.0f);
        glm::mat4 projection = GetProjectionMatrix(ScreenSize);
        const glm::mat4 m = glm::inverse(projection * modelView);
    
        // Need to invert Y since screen Y-origin point  down,
        // while 3D Y-origin points up (this is an OpenGL only requirement):
        winY = ScreenSize.cy - winY;
    
        // Transformation of normalized coordinates between -1 and 1:
        glm::vec4 in;
        in.x = winX  / ScreenSize.cx  * 2.0 - 1.0;
        in.y = winY  / ScreenSize.cy  * 2.0 - 1.0;
        in.z = 2.0 * winZ - 1.0;
        in.w = 1.0;
    
        // To world coordinates:
        glm::vec4 out(m * in);
        if (out.w == 0.0) // Avoid a division by zero
        {
            return glm::vec3(0.0f);
        }
        out.w = 1.0 / out.w;
        return glm::vec3(out.x * out.w, out.y * out.w,out.z * out.w);
    }

Since this function is basic rewrite of the pseudo code (from here) and I'm far from behind good at mathematics I don't really see what could go wrong...

PS: my view matrix (provided by GetViewMatrix()) is correct (since I use it to show my scene)

my projection matrix is also correct

the ScreenSize object carry my viewport size

score 1 · Accepted Answer · answered Aug 12 '20 at 00:32

I have found what's wrong, the return vec3 should be made by dividing each component by the perspective instead of being multiply by it. Here is the new UnProject function :

    glm::vec3 UnProject2(float winX, float winY,float winZ){
        glm::mat4 View = GetViewMatrix() * glm::mat4(1.0f);
        glm::mat4 projection = GetProjectionMatrix(ScreenSize);
        glm::mat4 viewProjInv = glm::inverse(projection * View);
        
        winY = ScreenSize.cy - winY;
        
        glm::vec4 clickedPointOnSreen;
        clickedPointOnSreen.x = ((winX - 0.0f) / (ScreenSize.cx)) *2.0f -1.0f;
        clickedPointOnSreen.y = ((winY - 0.0f) / (ScreenSize.cy)) * 2.0f -1.0f;
        clickedPointOnSreen.z = 2.0f*winZ-1.0f;
        clickedPointOnSreen.w = 1.0f;
        
        glm::vec4 clickedPointOrigin  =  viewProjInv * clickedPointOnSreen;
        return glm::vec3(clickedPointOrigin.x / clickedPointOrigin.w,clickedPointOrigin.y / clickedPointOrigin.w,clickedPointOrigin.z / clickedPointOrigin.w);
    }

I also changed the way start and end are calculated :

glm::vec3 start = UnProject2(x,y,0.0f);
glm::vec3 end = UnProject2(x,y,1.0f);

Issue with Picking (custom unProject() function)

1 Answers1