get mouse in world coordinates with 2 gluUnproject calls to create ray

Question

I try to use what many people seem to find a good way, I call gluUnproject 2 times with different z-values and then try to calculate the direction vector for the ray from these 2 vectors.

I read this question and tried to use the structure there for my own code:

        glGetFloat(GL_MODELVIEW_MATRIX, modelBuffer);
        glGetFloat(GL_PROJECTION_MATRIX, projBuffer);
        glGetInteger(GL_VIEWPORT, viewBuffer);

        gluUnProject(mouseX, mouseY, 0.0f, modelBuffer, projBuffer, viewBuffer, startBuffer);
        gluUnProject(mouseX, mouseY, 1.0f, modelBuffer, projBuffer, viewBuffer, endBuffer);

        start = vecmath.vector(startBuffer.get(0), startBuffer.get(1), startBuffer.get(2));
        end = vecmath.vector(endBuffer.get(0), endBuffer.get(1), endBuffer.get(2));

        direction = vecmath.vector(end.x()-start.x(), end.y()-start.y(), end.z()-start.z());

But this only returns the Homogeneous Clip Coordinates (I believe), since they only range from -1 to 1 on every axis.

How to actually get coordinates from which I can create a ray?

EDIT: This is how I construct the matrices:

Matrix projectionMatrix = vecmath.perspectiveMatrix(60f, aspect, 0.1f,
            100f);
//The matrix of the camera = viewMatrix
setTransformation(vecmath.lookatMatrix(eye, center, up));
//And every object sets a ModelMatrix in it's display method  
Matrix modelMatrix = parentMatrix.mult(vecmath
                    .translationMatrix(translation));
modelMatrix = modelMatrix.mult(vecmath.rotationMatrix(1, 0, 1, angle));

EDIT 2:

This is how the function looks right now:

private void calcMouseInWorldPosition(float mouseX, float mouseY, Matrix proj, Matrix view) {
    Vector start = vecmath.vector(0, 0, 0);
    Vector end = vecmath.vector(0, 0, 0);

    FloatBuffer modelBuffer = BufferUtils.createFloatBuffer(16);
    modelBuffer.put(view.asArray());
    modelBuffer.rewind();
    FloatBuffer projBuffer = BufferUtils.createFloatBuffer(16);
    projBuffer.put(proj.asArray());
    projBuffer.rewind();

    FloatBuffer startBuffer = BufferUtils.createFloatBuffer(16);
    FloatBuffer endBuffer = BufferUtils.createFloatBuffer(16);
    IntBuffer viewBuffer = BufferUtils.createIntBuffer(16);

     //The two calls for projection and modelView matrix are disabled here,   
       as I use my own matrices in this case
//  glGetFloat(GL_MODELVIEW_MATRIX, modelBuffer);
//  glGetFloat(GL_PROJECTION_MATRIX, projBuffer);
    glGetInteger(GL_VIEWPORT, viewBuffer);

    //I know this is really ugly and bad, but I know that the height and width is always 600  
    // and this is just for testing purposes 
    mouseY = 600 - mouseY;

    gluUnProject(mouseX, mouseY, 0.0f, modelBuffer, projBuffer, viewBuffer, startBuffer);
    gluUnProject(mouseX, mouseY, 1.0f, modelBuffer, projBuffer, viewBuffer, endBuffer);

    start = vecmath.vector(startBuffer.get(0), startBuffer.get(1), startBuffer.get(2));
    end = vecmath.vector(endBuffer.get(0), endBuffer.get(1), endBuffer.get(2));

    direction = vecmath.vector(end.x()-start.x(), end.y()-start.y(), end.z()-start.z());

}

I'm trying to use my own projection and view matrix, but this only seems to give weirder results.
With the GlGet... stuff I get this for a click in the upper right corner:
start: (0.97333336, -0.98, -1.0)
end: (0.97333336, -0.98, 1.0)

When I use my own stuff I get this for the same position:
start: (-2.4399707, -0.55425626, -14.202201)
end: (-2.4399707, -0.55425626, -16.198204)

Now I actually need a modelView matrix instead of just the view matrix, but I don't know how I am supposed to get it, since it is altered and created anew in every display call of every object.
But is this really the problem? In this tutorial he says "Normally, to get into clip space from eye space we multiply the vector by a projection matrix. We can go backwards by multiplying by the inverse of this matrix." and in the next step he multiplies again by the inverse of the view matrix, so I thought this is what I should actually do?

EDIT 3:
Here I tried what user42813 suggested:

    Matrix view = cam.getTransformation();
    view = view.invertRigid();

    mouseY = height - mouseY - 1;

    //Here I only these values, because the Z and W values would be 0  
    //following your suggestion, so no use adding them here
    float tempX = view.get(0, 0) * mouseX + view.get(1, 0) * mouseY;
    float tempY = view.get(0, 1) * mouseX + view.get(1, 1) * mouseY;
    float tempZ = view.get(0, 2) * mouseX + view.get(1, 2) * mouseY;

    origin = vecmath.vector(tempX, tempY, tempZ);
    direction = cam.getDirection();

But now the direction and origin values are always the same:
origin: (-0.04557252, -0.0020000197, -0.9989586)
direction: (-0.04557252, -0.0020000197, -0.9989586)

`gluUnProject` is *supposed* to go all the way back to object-space coordinates. More than likely you have an identity ModelView/Projection matrix. What version of GL are you targeting? I can easily see `glGetFloat (...)` returning uninitialized matrices if you're using GL 3.2 core, for instance. In that case, only the viewport would be defined and that would explain why you only go back to clip-space. — Andon M. Coleman, Sep 19 '14 at 17:53
Is `GL_ARB_compatibility` one of the extensions in your extensions string? Moreover, if you are constructing the matrices using a separate library, why are you trying to query them from OpenGL? — Andon M. Coleman, Sep 19 '14 at 19:17
@AndonM.Coleman Sorry I don't know what extension string means. And well I don't know how else I am supposed to get the required matrices for the gluUnproject method, especially the modelMatrix, since it is changed with every object. I could somehow get the view and projection matrices there, but I still need the ModelView matrix and how am I supposed to create/get it? — Big_Chair, Sep 19 '14 at 19:36
@AndonM.Coleman Yes, right now I tried doing modelBuffer.put(view.asArray()); projBuffer.put(proj.asArray()); But I can only use the view and projection matrix, I can't get the modelmatrix at this point. So I used the view matrix instead of the modelView. But this does not really seem to work..... — Big_Chair, Sep 19 '14 at 20:06
You're going to have to explain your situation a little clearer, unfortunately (in your question, because these comments are becoming a mess). I'm not even sure if you're using the fixed-function or shader pipeline at this point. But if you have the view matrix, you should just be able to multiply it by your object's model matrix to get the modelview matrix. In any case, if ***that*** much were actually working (*sans* model matrix), it would be in world-space not clip-space as you are describing :-\ — Andon M. Coleman, Sep 19 '14 at 20:28
@AndonM.Coleman Yes, I updated my question now, I hope this can clarify some things — Big_Chair, Sep 19 '14 at 21:49

score 1 · Accepted Answer · answered Sep 26 '14 at 19:09

Ok I finally managed to work this out, maybe this will help someone.
I found some formula for this and did this with the coordinates that I was getting, which ranged from -1 to 1:

    float tempX = (float) (start.x() * 0.1f * Math.tan(Math.PI * 60f / 360));
    float tempY = (float) (start.y() * 0.1f * Math.tan(Math.PI * 60f / 360) * height / width);
    float tempZ = -0.1f;

    direction = vecmath.vector(tempX, tempY, tempZ); //create new vector with these x,y,z
    direction = view.transformDirection(direction); 
    //multiply this new vector with the INVERSED viewMatrix
    origin = view.getPosition(); //set the origin to the position values of the matrix (the right column)

Mehdi · Answer 2 · 2014-09-19T22:20:58.043

0

I dont really use deprecated opengl but i would share my thought, First it would be helpfull if you show us how you build your View matrix, Second the View matrix you have is in the local space of the camera, now typically you would multiply your mouseX and (ScreenHeight - mouseY - 1) by the View matrix (i think the inverse of that matrix sorry, not sure!) then you will have the mouse coordinates in camera space, then you will add the Forward vector to that vector created by the mouse, then you will have it, it would look something like that:

float mouseCoord[] = { mouseX, screen_heihgt - mouseY - 1, 0, 0 }; /* 0, 0 because we multipling by a matrix 4.*/
mouseCoord = multiply( ViewMatrix /*Or: inverse(ViewMatrix)*/, mouseCoord );
float ray[] = add( mouseCoord, forwardVector );

edited Sep 19 '14 at 22:20

answered Sep 19 '14 at 18:00

Mehdi

582
4
14

what of my code is deprecated? The glGetxxx() methods? What would be modern ways of getting that? – Big_Chair Sep 19 '14 at 18:41
the gluPerspective(..), gluUnProject(...), glMatrixMode(..), glTranslateX(..), glRotateX(..), ..., well, the modern way will be creating your own math library, or using an already defined one, such as glm if you are using C/C++ or take a look at LibGDX math lib it's a greate one, you also have to use shaders without the gl_ModelViewMatrix/gl_NormalMatrix/gl_Vertex ..., and the list goes one, you can have some great tuts here [link](http://www.opengl-tutorial.org/), also note that if the code i provide works, then the ray would be in camera space. – Mehdi Sep 19 '14 at 22:30
about the code you provide it's note a good idea to create directe buffers every frame, what you should do, is create the static refrences and reuse them. – Mehdi Sep 19 '14 at 22:31
One more thing, if you are using the mouse to control the camera rotation, then the code simplifies to just ( ray = cameraForward ). – Mehdi Sep 19 '14 at 22:34
I edited my question and tried what you suggested. but it didnt really work :/ – Big_Chair Sep 20 '14 at 12:22

get mouse in world coordinates with 2 gluUnproject calls to create ray

2 Answers2

Linked