Ray picking with gluUnProject() into world coordinates

Question

I set up a scene using:

glViewport(0,0,screen->w,screen->h);
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
const float wh = (float)screen->w/(float)screen->h;
gluPerspective(zoom,wh,1,10);
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glTranslatef(0,0,-3); // puts model inside z-range

Later, when the user clicks, I want to calculate a ray in world coordinates for my intersection test (not using glReadPixels() to get the z):

double mv[16], p[16], a, b, c, d, e, f;
glGetDoublev(GL_MODELVIEW_MATRIX,mv);
glGetDoublev(GL_PROJECTION_MATRIX,p);
matrix_t _mv, _p;
glGetFloatv(GL_MODELVIEW_MATRIX,_mv.f);
glGetFloatv(GL_PROJECTION_MATRIX,_p.f);
const matrix_t inv = (_p*_mv).inverse();
GLint viewport[4];
glGetIntegerv(GL_VIEWPORT,viewport);
gluUnProject(x,viewport[3]-y,0,mv,p,viewport,&a,&b,&c);
const vec_t origin(vec_t(a,b,c)*inv);
gluUnProject(x,viewport[3]-y,1,mv,p,viewport,&d,&e,&f);
const vec_t dest(vec_t(d,e,f)*inv);
ray_t ray(origin,dest-origin);

Unfortunately I misunderstand how to convert the ray from camera coordinates to world coordinates - my ray misses.

How do you correctly convert the ray to world coordinates?

Answer 1

You might find my answer to a similar question useful.

Click to zoom in WebGL

That answer pertains to WebGL and Javascript. I don't work in C++ and OpenGL. but looking at your code I would think that gluUnproject is already using the inverses of the p and mv matrices. You multiplication of the result by inv is therefore redundant, and is probably what is screwing up your result.