8

I'm trying to port an old Augmented Reality project, which is based on the metaio framework for an other see-through HMD.

Cause metaio is bought by apple, it's available documentation and support is very limited and this new HMD is not supported out of the box.

Everything is already working, besides the camera calibration / hand eye calibration.

I have already calculated a Projection matrix P (using SPAAM) which transforms points in the camera coordinate system to the screen (Tested by custom rendering). Unfortunately I can't set the projection matrix directly in metaio but have to separately set the camera parameters and Hand Eye calibration for rendering.

Therefore I have extracted the extrinsic and intrinsic camera parameters from the Projection matrix P as described here.

But those are not working correctly when set. The result seems to be not completely off, cause the rendered scale seems to be correct and manual measured distance between camera and eye is about 126mm (translation on x axis).

Projection matrix P: Projection matrix P

Intrinsics (resolution: 1280 x 720):

intrinsics

Extrinsics (hec?): extrinsics

I guess I'm missing some additional step I'm not aware of. I appreciate any ideas and help.

Additional information:

  • metaio is using right handed coordinate systems with camera looking in negative z axsis, x to the right and y up
  • metaio documentaion for hand eye calibration states: Transformation from camera to eye (display) coordinates
  • Since The translation part of HEC on the x- axis has to be positive (camera is on the right side of the HMD) I assume it's transformation is specified from eye to camera?
Zermingore
  • 743
  • 2
  • 11
  • 28
user2479595
  • 339
  • 2
  • 11
  • Do you not see your augmentations at all or do they show up in the wrong place, or at the wrong scale? Does the aspect ratio of your display also matches the 16:9 of the camera feed? If not you have to adapt the projection. – Joan Charmant Mar 09 '17 at 11:32
  • I see the augmentation, but not at the right spot. The aspect ratio and the size (combination of focal length and translation part on the z axis) seem to be correct. I'm using the same resolution for the camera and the display atm. I guess my hand eye calibration is not correct (Both translation and rotation). – user2479595 Mar 09 '17 at 16:33

1 Answers1

2

Does Metaio automatically invert your matrices for you? Because if it doesn't, this could be your problem. As y=0 is at the top of the screen in computer graphics many of these image/camera processing apps come with an option to invert the transformation matrix for you. If it doesn't you may want to just try manually inverting your transformation matrix to see if that works instead.

Rbit
  • 231
  • 1
  • 7
  • Unfortunately I don't know how metaio is handling the matrices. Inverting the extrinsics did not fix it. – user2479595 Mar 15 '17 at 13:37
  • Just to be absolutely clear - you didn't just flip the matrix over and try it: you ran through the inversion process for a matrix? (the wolfram alpha computation website can do this for you eliminating any calculation). I had hoped that would resolve the problem since your scaling along the z-axis works – Rbit Mar 15 '17 at 13:41
  • I get 0.00029414973499446282, 0, -0.07417029119926943 for the first row 0, 0.0003022967744552130, -0.2128135354974268 for the middle row and 0,0,1 for the final row – Rbit Mar 15 '17 at 13:52
  • I'm surprised that your result is 3x3. In order to inverse the matrix (E) I'm always adding a 0,0,0,1 row so I get 4x4 instead of 3x4 and then I'm calling the inverse function of my Matrix package (jama). And then I'm removing the 0,0,01 row again to get my inverse 3x4 matrix. – user2479595 Mar 15 '17 at 17:18
  • I just realized that you were inverting the intrinsics. I will try this out. So far I only tried inverting the extrinsics. – user2479595 Mar 15 '17 at 17:21
  • Ah yes! Sorry - I should have made that clearer. Trying to invert a non-square (rectangular shaped) matrix will not work as only square matrices have an inverse. – Rbit Mar 15 '17 at 17:41
  • Inverting the intrinsic seems not correct. Focal length and principal point are then way too small. From working samples I know that e.g. the principal point should be roughly in the middle. – user2479595 Mar 15 '17 at 21:10
  • In that case, and not being able to invert the other matrices as they are not square, I am afraid I am out of ideas. – Rbit Mar 15 '17 at 22:00