Convert a 2D screen coordinate to a 3D world coordinate

Question

Right now I can transform a 3D world coordinate to a 2D screen coordinate. I can achieve that with this:

    D3DXMATRIX viewMatrix = renderer->viewMatrix;
    D3DXMATRIX projectionMatrix = renderer->projectionMatrix;
    D3DXMATRIX viewProjectionMatrix = D3DXMATRIX();
    D3DXMatrixMultiply(&viewProjectionMatrix, &viewMatrix, &projectionMatrix);

    D3DXVECTOR3 pScreen = D3DXVECTOR3(0.0f, 0.0f, 0.0f);
    D3DXVECTOR3 pWorld(world->getX(), world->getY(), world->getZ());
    D3DXVec3TransformCoordImpl(&pScreen, &pWorld, &viewProjectionMatrix);

Now pScreen contains the 2D screen coordinate

How could I reverse this to get a 3D world coordinate out of a 2D screen coordinate? I'm using this implementation of the D3DXVec3TransformCoord function:

D3DXVECTOR3* WINAPI D3DXVec3TransformCoordImpl(D3DXVECTOR3* pout, CONST D3DXVECTOR3* pv, CONST D3DXMATRIX* pm)
{
    FLOAT norm = pm->m[0][3] * pv->x + pm->m[1][3] * pv->y + pm->m[2][3] * pv->z + pm->m[3][3];
    if (norm)
    {
        pout->x = (pm->m[0][0] * pv->x + pm->m[1][0] * pv->y + pm->m[2][0] * pv->z + pm->m[3][0]) / norm;
        pout->y = (pm->m[0][1] * pv->x + pm->m[1][1] * pv->y + pm->m[2][1] * pv->z + pm->m[3][1]) / norm;
        pout->z = (pm->m[0][2] * pv->x + pm->m[1][2] * pv->y + pm->m[2][2] * pv->z + pm->m[3][2]) / norm;
    }
    else
    {
        pout->x = 0.0f;
        pout->y = 0.0f;
        pout->z = 0.0f;
    }
    return pout;
}

Do you know on what z plane the point is on or any other way to determine z value? Because you need multiple points — Jake Freeman, Jan 03 '18 at 12:57
@JakeFreeman I have a function returns every z value for every 3D x and y coordinate. So it's like a map. — Bob A, Jan 03 '18 at 13:01
See this point you need to look at lots of points at once: https://stackoverflow.com/questions/76134/how-do-i-reverse-project-2d-points-into-3d — Jake Freeman, Jan 03 '18 at 13:03
A 2D point on the screen corresponds to a *line* in 3D space, passing through the camera's location. You would need to specify a Z-value to obtain a unique point. — meowgoesthedog, Jan 03 '18 at 13:40

score -1 · Answer 1 · answered Jan 03 '18 at 13:13

You could use the SolvePnP library like this:

int max = std::max(img.rows,img.cols);
Mat camMatrix = (Mat_<double>(3,3) <<
    max_d,   0, img.cols/2.0,
    0,   max_d, img.rows/2.0,
    0,   0,     1.0);

// 2d -> 3d correspondence
vector<Point2d> pts2d = ...
vector<Point3d> pts3d = ...
Mat rvec,tvec;
solvePnP(pts3d, pts2d, camMatrix, Mat(1,4,CV_64F,0.0), rvec, tvec, false, SOLVEPNP_EPNP);
// get 3d rot mat
Mat rotM(3, 3, CV_64F);
Rodrigues(rvec, rotM);

// push tvec to transposed Mat
Mat rotMT = rotM.t();
rotMT.push_back(tvec.reshape(1, 1));

// transpose back, and multiply
camMatrix * rotMT.t();

Convert a 2D screen coordinate to a 3D world coordinate

1 Answers1