3d Math : screen space to world space

Question

I've been trying to implement clicking in my webgl app for the last 6 hours and I can't find anything clear enough about this subject.

What I have came up with so far is, in pseudo code:

screenSpace = mousePosition;
normalizedScreenSpace = (screenSpace.x/screen.width, screenSpace.y/screen.height);

camSpace = invertProjectionMatrix * normalizedScreenSpace;

worldSpace = invertViewMatrix * camSpace;

Printing out the worldSpace coordinates, and it doesn't corresponds to other objects in the scene. What am I doing wrong?

Possible duplicate of [How to get object in WebGL 3d space from a mouse click coordinate](http://stackoverflow.com/questions/7364693/how-to-get-object-in-webgl-3d-space-from-a-mouse-click-coordinate) — LJᛃ, Feb 23 '16 at 17:21
@LJ Answers from this question are unsatisfying. One links to his implementation with awefully bad nomenclature and commentary which makes it pointless if you don't already understand it. Or a paper of a very high level review of the subject. And a Three.js version which doesn't answer the question at all. — Henri Latreille, Feb 23 '16 at 17:31
The accepted answer links to [the glu unproject implementation](https://www.opengl.org/wiki/GluProject_and_gluUnProject_code) which imho is as descriptive as it gets. Your normalization is in 0-1 range whereas it should be -1,+1 in addition to that you seem to not apply the perspective divide. — LJᛃ, Feb 23 '16 at 17:50

WacławJasper · Accepted Answer · 2016-02-24T17:27:41.037

2

The viewProjection matrix brings a vec3 from world space to clip space and so its inverse does the reverse, clip space to world space. Whats missing is the perspective divide that gpu handles for you behind the hood so you have to account for that as well. Add in the screen width and height and you have your screen to world:

screenToWorld: function(invViewProjection, screenWidth, screenHeight){
    // expects this[2] (z value) to be -1 if want position at zNear and +1 at zFar

    var x = 2*this[0]/screenWidth - 1.0;
    var y = 1.0 - (2*this[1]/screenHeight); // note: Y axis oriented top -> down in screen space
    var z = this[2];
    this.setXYZ(x,y,z);
    this.applyMat4(invViewProjection);
    var m = invViewProjection;
    var w = m[3] * x + m[7] * y + m[11] * z + m[15]; // required for perspective divide
    if (w !== 0){
        var invW = 1.0/w;
        this[0] *= invW;
        this[1] *= invW;
        this[2] *= invW;
    }

    return this;
},

And the reverse calculation:

worldToScreen: function(viewProjectionMatrix, screenWidth, screenHeight){
    var m = viewProjectionMatrix;
    var w = m[3] * this[0] + m[7] * this[1] + m[11] * this[2] + m[15]; // required for perspective divide
    this.applyMat4(viewProjectionMatrix);
    if (w!==0){ // do perspective divide and NDC -> screen conversion
        var invW = 1.0/w;
        this[0] = (this[0]*invW + 1) / 2 * screenWidth;
        this[1] = (1-this[1]*invW) / 2 * screenHeight; // screen space Y goes from top to bottom
        this[2] *= invW;
    } 
    return this;
},

edited Feb 24 '16 at 17:27

answered Feb 23 '16 at 23:32

WacławJasper

3,284
14
19

Now this is a great quality answer. I have more 2 questions : when you multiply the initial XYZ with the inverted view projection matrix, what is the value of W? Is it assumed to be 0? – Henri Latreille Feb 24 '16 at 14:15
My second question is, if I would want to get a world coord to screen coord like a shader does but in CPU, I suppose I would do modelViewProjMatrix * vec4(vector,1). But then, how is the perspective divide computed? (I tried dividing by the w part of the resulting vector but my numbers are off) – Henri Latreille Feb 24 '16 at 14:21
1

The W is assumed to be 1. What ought to be happening is that I convert to Vec4(x,y,z,1) first then do the `applyMat4` on vec4. Note that `var w = m[3] * x + m[7] * y + m[11] * z + m[15]` should be `+W*m[15]` but W is implicitly assumed to be 1. I updated answer for the world to screen. – WacławJasper Feb 24 '16 at 17:29
I have a question about the screenToWorld raycasting. I need the points from znear and zfar to cast a ray and test it against the triangles in the scene, but what if all my triangles are at z = 0? Is it possible to call screenToWorld once to get the position in world space at z = 0 or do I have to get a ray and translate on this ray to z = 0 to get the coordinate? Thank you – Henri Latreille Feb 25 '16 at 15:54
I think you are confusing some things. What you want to do is to get the 2 points of the ray segment in world position. This can be done by setting the vec3 to (mouse.x, mouse.y, 1) then screenToWorld for pt A and (mouse.x, mouse.y, -1) for pt B. Now you have ray segment AB in world position. Then bring your triangle and the ray into the same coordinate space and run the ray-triangle test. – WacławJasper Feb 26 '16 at 02:31

3d Math : screen space to world space

1 Answers1

Linked