2

I'm tracking a marker with ARToolKit+. I receive a model view matrix that looks about right. Now I'd like to warp the image in a way that the marker looks just like it would look if I looked straight at it. But whatever I do, the result looks just extremely distorted. I know that ARToolKit stores the 4x4 matrix in column major order, so I fixed that for OpenCV.

What I tried so far was: 1) fix the order to row major order 2) calculate the inverse with cvInverse (although transposing the 3x3 rotation part + inverting the translation should suffice) 3) use that matrix with cvPerspectiveWarp

Am I doing something wrong?

tl;dr: I want this: https://www.youtube.com/watch?v=qZ-LU-C2p2Q I get some distorted lines and lots of black instead.

Wooble
  • 87,717
  • 12
  • 108
  • 131
  • I imagine your problem is in going from the 4x4 matrix from ARTK to the 3x3 for warp perspective. How are you doing it? – Hammer Dec 06 '12 at 22:42
  • As far as I know the last column is the x / y / z translation, and the last row is always 0 0 0 1, 1 for the homogenous coordinate. Right now I simply strip those away, and just keep the "scaling" and rotation part... while the scaling should be 1.0. –  Dec 07 '12 at 05:15

1 Answers1

2

Your problem is in converting from 4x4 to 3x3. The short answer is that you want to drop the 3rd column and bottom row to make the 3x3 and then premultiply with your camera matrix. For a longer explanation see here

Clarification The pose you get from ARTK represents a transform from one place to another. When I say "the initial image appears without rotation" I meant that your transform goes from an initial state which has no rotation about the x or y axis to the current state. That is a fine assumption for most augmented reality applications, I mentioned it just to be thorough.

As for why you can drop the 3rd column. Since you are transforming a plane, your z coordinate can be completely expressed by your x and y coordinates given the equation of your plane. If we assume that initially there is no rotation then your initial z coordinate is a constant value. If there is rotation then z is not constant but it varies deterministically in x and y according to its plane equation which can still be expressed in one matrix (though you don't need that). Since in your case your 4x4 transform is probably expressing the transform from the marker lying flat at z = 0 to its current position, the 3rd column of your 4x4 matrix does nothing (it all gets multiplied by 0) so it can be dropped without affecting the result.

In short: Forget about the rotation stuff, its more complicated than you need, just realize that the transform is from initial coordinates to final coordinates and your initial coordinates are always

[x,y,0,1]

which makes your third column irrelevant.

Update I'm sorry! I just re-read your question and realized you just want to warp the marker so it looks like a straight on view, I got caught up in describing a general transform from 4x4 to 3x3. The 4x4 transform you get from ARTK is not the transform that will de warp the warker, it is the transform that moves the marker from the origin to its final position. To de warp the marker like you asked the process is similar but would be slightly different. I haven't done that before but here is my guess.

First, you need to get the 4x4 transform between where the marker is in world space, and where you would like it to appear to be after warping it. Right now the transform goes from the origin to the marker location. To change the transform to go from some point farther down on the z axis (say 100) to the marker location define the transform.

initial_marker_pose = [1,0,0,0
                       0,1,0,0
                       0,0,1,100
                       0,0,0,1];

Now you have the transform from the origin to what you want as your "inital" position, and the transform from the origin to your "final" position. To get the transform from initial to final simply

initial_to_final = origin_to_marker*initial_marker_pose.inv();

Now you would follow the process outlined in the link I gave you, in this case your initial zpos is no longer 0, it is 100. Then when you are finished you will need to invert your 3x3 matrix. That is because this process takes you from a straight on view to the one defined by the pose from ARTK and you want the opposite of that. You will need to experiment with the initial z position. The smaller it is, the larger your marker will appear after de-warping.

Hopefully that works, sorry for the confusion about your question.

Community
  • 1
  • 1
Hammer
  • 10,109
  • 1
  • 36
  • 52
  • Thanks, I looked at your long answer, but I didn't really get all of it. If I drop the third column, won't that take away parts of the rotation? Also, I did not really understand what you mean by "If you need the solution to work without the assumption that the initial image appears without rotation let me know" -- the marker in the image can be rotated along all three axis. Do you mean the full image itself or the marker inside it? –  Dec 07 '12 at 07:03
  • Thank you for your updated answer, it helped a little. I tried implementing it, and the results are better I think, but still a lot of distortion happens. I do the following: I create a matrix ("converted", from your long answer with zpos = 0), invert it, and multiply it with my camera matrix. Is there still something wrong? –  Dec 10 '12 at 14:05
  • @LaurelWolf the only thing you should be inverting is the camera matrix. The result from the long answer is camera_mat*converted*projected – Hammer Dec 10 '12 at 15:45
  • Sorry, I misunderstood your post. Though, I do as you say now, and all I ever get is a single pixel stretched to the new image. :[ I have meassured the pixel coordinates of the marker myself, and had opencv create a perspective transform for me, in order to compare my output with what would be correct (i used the corners of the marker as input). Result for OpenCV's perspective transform matrix: 4.17 -0.3 -0. 0.05 3.09 -540 -0. -0. 1. My result: -0. 0. 0.5 -.0 0.0 -0.4 -0.0 -0. 0.1 (when I write 0. i mean it's not exactly zero, but close). Any ideas? –  Dec 11 '12 at 10:28
  • If it helps: camera matrix is like this: 534 0 309 0 534 214 0 0 1. projection matrix is: 0.002 -3.5e-10 -0.6 -6.5e-10 0.002 -0.4. the converted matrix is: -0.986 -0.071 26 0.0578 0.994 11.1 -0.156 -0.08 342 –  Dec 11 '12 at 10:51