OpenCV solvePnP to OpenGL: Large translation vector

Question

I'm building an Mac OSX AR application using OpenCV and OpenGL. It is marker-based AR and I'm using a library called ARma for that. This lib just writes the visual output directly to the image buffer using OpenCV functions. But I want to use OpenGL for visual output so I created an OpenGL View which I'm initializing by using the intrinsic parameters (which I got from using an OpenCV cam calibration program) for the GL_PROJECTION matrix as follows:

cv::Mat intrMat = tracker.camMatrix; // intrinsic 3x3 matrix
// contains:
// 641.68  0.00    319.50   
// 0.00    641.68  212.50   
// 0.00    0.00    1.00

const float fx = intrMat.at<double>(0,0);
const float fy = intrMat.at<double>(1,1);
const float fovy = 2 * atan(0.5f * screenH / fy) * RAD2DEG;
const float aspect = (screenW*fy) / (screenH*fx);
// GL Perspective values: fx=641.68 / fy=641.68 / fovy=41.01 / aspect=1.33

gluPerspective(fovy, aspect, 0.1, 100.0);

This is how I've seen it in similar projects and I think until here everything's correct. When a marker is detected, the extrinsic parameters are calculated in the ARma library using cvFindExtrinsicCameraParams2() (which is the same as cv::solvePnP). So I get the rotation vector rotVec and the translation vector tVec and throw them into the following method to calculate the GL_MODELVIEW matrix:

-(void)calcModelviewMat:(GLfloat *)mvMat fromRotVec:(cv::Mat const &)rotVec transVec:(cv::Mat const &)tVec {
    cv::Mat rotMat;
    cv::Mat rotVecGL = rotVec.clone();

    rotVecGL.at<float>(1) = -rotVecGL.at<float>(1);
    rotVecGL.at<float>(2) = -rotVecGL.at<float>(2);

    cv::Rodrigues(rotVecGL, rotMat);

    // init array with zeros
    memset((void *)mvMat, 0, 16 * sizeof(float));

    // set the rotation
    for (int i = 0; i < 3; i++) {
        for (int j = 0; j < 3; j++) {
            mvMat[i * 4 + j] = rotMat.at<float>(i, j);
        }
    }

    // set the translation
    for (int i = 0; i < 3; i++) {
        float s = (i > 0) ? -1 : 1;
        mvMat[12 + i] = s * tVec.at<float>(i);
    }

    mvMat[15] = 1;
}

This is the way of transforming these two vectors into a GL_MODELVIEW matrix that I've found here. I believe that this is correct but the problem is that the components in the translation vector which I get are much too large (between +/-500 and +/-1500) to get displayed. Here is an example output of the matrix:

0.96    0.05    -0.26   0.00    
0.01    0.97    0.23    0.00    
0.27    -0.22   0.94    0.00    
-72.55  8.47    -479.63 1.00

Why is that so? What kind of units does solvePnP() use in the translation vector and how can I transform them into usable values for displaying in my OpenGL View?