5

I try to match two overlapping images captured with a camera. To do this, I'd like to use OpenCV. I already extracted the features with the SurfFeatureDetector. Now I try to to compute the rotation and translation vector between the two images.

As far as I know, I should use cvFindExtrinsicCameraParams2(). Unfortunately, this method require objectPoints as an argument. These objectPoints are the world coordinates of the extracted features. These are not known in the current context.

Can anybody give me a hint how to solve this problem?

SecStone
  • 1,733
  • 4
  • 20
  • 31

2 Answers2

9

The problem of simultaneously computing relative pose between two images and the unknown 3d world coordinates has been treated here:

Berthold K. P. Horn. Relative orientation revisited. Berthold K. P. Horn. Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 545 Technology ...

EDIT: here is a link to the paper: http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.64.4700

Please see my answer to a related question where I propose a solution to this problem:

OpenCV extrinsic camera from feature points

EDIT: You may want to take a look at bundle adjustments too,

http://en.wikipedia.org/wiki/Bundle_adjustment

That assumes an initial estimate is available.

EDIT: I found some code resources you might want to take a look at:

Resource I:

http://www.maths.lth.se/vision/downloads/

Two View Geometry Estimation with Outliers

C++ code for finding the relative orientation of two calibrated cameras in presence of outliers. The obtained solution is optimal in the sense that the number of inliers is maximized.

Resource II:

http://lear.inrialpes.fr/people/triggs/src/ Relative orientation from 5 points: a somewhat more polished C routine implementing the minimal solution for relative orientation of two calibrated cameras from unknown 3D points. 5 points are required and there can be as many as 10 feasible solutions (but 2-5 is more common). Also requires a few CLAPACK routines for linear algebra. There's also a short technical report on this (included with the source).

Resource III:

http://www9.in.tum.de/praktika/ppbv.WS02/doc/html/reference/cpp/toc_tools_stereo.html vector_to_rel_pose Compute the relative orientation between two cameras given image point correspondences and known camera parameters and reconstruct 3D space points.

Community
  • 1
  • 1
Rulle
  • 4,496
  • 1
  • 15
  • 21
2

There is a theoretical solution, however, the OpenCV implementation of camera pose estimation lacks the needed tools.

The theoretical approach: Step 1: extract the homography (the matrix describing the geometrical transform between images). use findHomography() Step 2. Decompose the result matrix into rotations and translations. Use cv::solvePnP();

Problem: findHomography() returns a 3x3 matrix, corresponding to a projection from a plane to another. solvePnP() needs a 3x4 matrix, representing the 3D rotation/translation of the objects. I think that with some approximations, you can modify the solvePnP to give you some results, but it requires a lot of math and a very good understanding of 3D geometry.

Read more about at http://en.wikipedia.org/wiki/Transformation_matrix

Sam
  • 19,708
  • 4
  • 59
  • 82
  • The images will be related by a homography if (i) The only difference in pose is rotation about the camera centre _or_ (ii) Both images depict a planar object filling the whole image plane. In general, a homography does not model the transformation between the images. – Rulle Nov 20 '11 at 18:58
  • What you describe here is an affine transform. A homography is a general transform from any 3D position to any other 3D position. It is completely described by a 4x4 matrix. However, if you apply some constrains to the transform (like all of them should stay in the same plane) you can use only a 3x3 matrix. – Sam Nov 20 '11 at 19:30
  • Oh, I thought you meant the 3x3 homography matrix that relates 2d image points represented with homogenous coordinates between the two images. – Rulle Nov 20 '11 at 19:47
  • Homography assumes that all points lie on the same plane, which might not be the case here. As for the rotation and translation - you can compute the fundamental matrix (cv::findFundamentalMat()), calculate the essential matrix (cv::findEssentialMat()) and decompose the second using SVD plus a couple of additional steps (see book "Multiple View Geometry in Computer Visions" by Hartley and Zisserman) in order to get those. – rbaleksandar May 22 '14 at 12:25
  • Homography will only work if you are testing points on a plane. Or the motion is purely rotational. Otherwise you will have to use the essential matrix to extract the relative pose – Yonatan Simson May 24 '17 at 09:21