Unable to mouse pick a quad rendered in a framebuffer

Question

Struggling to mouse pick a point/quad. I believe I am either using coordinates in the wrong space or perhaps not accounting for the framebuffer's position/size (it's a sub window of the main window).

Tried converting to various different coordinate spaces and inverting the model matrix too. Currently projecting a ray in world space (hopefully correctly) and trying to compare it to the point's (quad) location. The point is specified in local space, but the entity is rendered at the origin (0f, 0f, 0f) therefore I don't think it should be any different in world space?

To get the mouse ray in world space:

private fun calculateRay(): Vector3f {
        val mousePosition = Mouse.getCursorPosition()
        val ndc = toDevice(mousePosition)
        val clip = Vector4f(ndc.x, ndc.y, -1f, 1f)
        val eye = toEye(clip)
        return toWorld(eye)
    }

private fun toDevice(mousePosition: Vector2f): Vector2f {
        mousePosition.x -= fbo.x // Correct thing to do?
        mousePosition.y -= fbo.y
        val x = (2f * mousePosition.x) / fboSize.x - 1
        val y = (2f * mousePosition.y) / fboSize.y - 1
        return Vector2f(x, y)
    }

private fun toEye(clip: Vector4f): Vector4f {
        val invertedProjection = Matrix4f(projectionMatrix).invert()
        val eye = invertedProjection.transform(clip)
        return Vector4f(eye.x, eye.y, -1f, 0f)
    }

    private fun toWorld(eye: Vector4f): Vector3f {
        val viewMatrix = Maths.createViewMatrix(camera)
        val invertedView = Matrix4f(viewMatrix).invert()
        val world = invertedView.transform(eye)
        return Vector3f(world.x, world.y, world.z).normalize()
    }

When hovering over a point (11.25, -0.75), the ray coords are (0.32847548, 0.05527423). I tried normalising the point's position and it is still not a match.

Feel like I am missing/overlooking something or just manipulating the coordinate systems incorrectly. Any insight would be greatly appreciated, thank you.

EDIT with more information:

The vertices of the quad are: (-0.5f, 0.5f, -0.5f, -0.5f, 0.5f, 0.5f, 0.5f, -0.5f)

Loading matrices to shader:

private fun loadMatrices(position: Vector3f, rotation: Float, scale: Float, viewMatrix: Matrix4f, currentRay: Vector3f) {
        val modelMatrix = Matrix4f()
        modelMatrix.translate(position)
        modelMatrix.m00(viewMatrix.m00())
        modelMatrix.m01(viewMatrix.m10())
        modelMatrix.m02(viewMatrix.m20())
        modelMatrix.m10(viewMatrix.m01())
        modelMatrix.m11(viewMatrix.m11())
        modelMatrix.m12(viewMatrix.m21())
        modelMatrix.m20(viewMatrix.m02())
        modelMatrix.m21(viewMatrix.m12())
        modelMatrix.m22(viewMatrix.m22())
        modelMatrix.rotate(Math.toRadians(rotation.toDouble()).toFloat(), Vector3f(0f, 0f, 1f))
        modelMatrix.scale(scale)
        shader.loadModelViewMatrix(viewMatrix.mul(modelMatrix))
        shader.loadProjectionMatrix(projectionMatrix)
    }

Calculating gl_Position in vertex shader:

gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 0.0, 1.0);

EDIT 2: Altered my code after reading up some more material based on Rabbid's comments. Not sure if I require the division by 2 in the viewport size (I have a retina MacBook display).

mousePosition.sub(fboPosition)
        val w = (fboSize.x / 2).toInt()
        val h = (fboSize.y / 2).toInt()
        val y = h - mousePosition.y

        val viewMatrix = Maths.createViewMatrix(camera)
        val origin = Vector3f()
        val dir = Vector3f()
        Matrix4f(projectionMatrix).mul(viewMatrix)
                                  .unprojectRay(mousePosition.x, y, intArrayOf(0, 0, w, h), origin, dir)

score 2 · Accepted Answer · answered Jun 24 '19 at 04:38

2

The top left origin of window sapce is (0,0). So if you get (0, 0), if the mouse is in the the top left of the window you've to skip:

mousePosition.x -= fbo.x // Correct thing to do?
mousePosition.y -= fbo.y

Since the bottom left of the framebuffer is (0,0), the y coordinate has to be flipped:

val y = 1 - (2f * mousePosition.y) / fboSize.y

When a Cartesian coordinate is transformed by a (inverse) projection matrix, then the result is a Homogeneous coordinates. You've to do a Perspective divide, to get a Cartesian coordinate in view space:

val eye = invertedProjection.transform(clip)
return Vector3f(eye.x/eye.w, eye.y/eye.w, eye.z/eye.w)

answered Jun 24 '19 at 04:38

Rabbid76

202,892
27
131
174

Thank you for your response. 1) I am using a UI library, so my framebuffer is positioned in a widget located at (174f, 5f), therefore I believe I should offset the mouse position? 2) I don't think this is needed as I am using LWJGL? (see: https://youtu.be/DLKN0jExRIM?t=397) 3) I did not do perspective division as "we only need to un-project the x,y part, so let's manually set the z,w part to mean 'forwards, and not a point'" (see: http://antongerdelan.net/opengl/raycasting.html). Updated my post with some more info. – Jack Jun 24 '19 at 15:28
Actually, after doing some more research, it looks like you are correct with your second point. I am using a different mouse handler to the one mentioned in the video, which has the y origin at the top left instead of the bottom left. – Jack Jun 24 '19 at 16:22
1

@Jack "*I did not do perspective division as "we only need to un-project the x,y part*" - Nonsens, in perspective projection the x, y component depends on the depth. Different depth (z coordinate) results in different x, y coordinates. At Perspective Projection the projection matrix describes the mapping from 3D points in the world as they are seen from of a pinhole camera, to 2D points of the viewport. You can't skip the perspective divide. – Rabbid76 Jun 24 '19 at 16:34
1

@Jack See [How to recover view space position given view space depth value and ndc xy](https://stackoverflow.com/questions/11277501/how-to-recover-view-space-position-given-view-space-depth-value-and-ndc-xy/46118945#46118945) and [Translation to world coordinates](https://stackoverflow.com/questions/50351326/translation-to-world-coordinates/50352124#50352124) – Rabbid76 Jun 24 '19 at 16:41
Thank you again for the comment. Yes, I thought that was strange but it was mentioned in the tutorial I linked so I followed it. Regardless, I have now updated my code with some new material based on what you have mentioned; does it look okay to you? I'm not sure how I can actually check whether it is working correctly. The next step is then to find when it intersects the translated quad... this seems best to do with the Intersectionf library. I tried using intersectRayAab() to no avail - if I have a point (11.25, -0.75, 26.25), what should my bounding box be? Thank you once again. – Jack Jun 24 '19 at 20:17
Okay, I solved it. Had to invert the y-coordinate, and then I set the bounding boxes to +/- the scale of the point. Thanks for all your help! – Jack Jun 24 '19 at 23:23

Unable to mouse pick a quad rendered in a framebuffer

1 Answers1