95

I am developing an Augmented Reality SDK on OpenCV. I had some problems to find tutorials on the topic, which steps to follow, possible algorithms, fast and efficient coding for real-time performance etc.

So far I have gathered the next information and useful links.

OpenCV installation

Download latest release version.

You can find installation guides here (platforms: linux, mac, windows, java, android, iOS).

Online documentation.

Augmented Reality

For begginers here is a simple augmented reality code in OpenCV. It is a good start.

For anyone searching for a well designed state-of-the-art SDK I found some general steps that every augmented-reality based on marker tracking should have, considering OpenCV functions.

  1. Main program: creates all classes, initialization, capture frames from video.

  2. AR_Engine class: Controls the parts of an augmented reality application. There should be 2 main states:

    • detection: tries to detect the marker in the scene
    • tracking: once it is detected, uses lower computational techniques for traking the marker in upcoming frames.

Also there should be some algorithms for finding the position and orientation of the camera in every frame. This is achieve by detecting the homography transformation between the marker detected in the scene, and a 2D image of the marker we have processed offline. The explanation of this method here (page 18). The main steps for Pose Estimations are:

  1. Load camera Intrinsic Parameters. Previously extracted offline through calibration. intrinsic parameters

  2. Load the pattern (marker) to track: It is an image of the planar marker we are going to track. It is necessary to extract features and generate descriptors (keypoints) for this pattern so later we can compare with features from the scene. Algorithms for this task:

  3. For every frame update, run a detection algorithm for extracting features from the scene and generate descriptors. Again we have several options.

    • SIFT
    • FAST
    • SURF
    • FREAK: A new method (2012) supossed to be the fastest.
    • ORB

  4. Find matches between pattern and the scene descriptors.

  5. Find Homography matrix from those matches. RANSAC can be used before to find inliers/outliers in the set of matches.

  6. Extract Camera Pose from homography.

Complete examples:

Community
  • 1
  • 1
Jav_Rock
  • 22,059
  • 20
  • 123
  • 164

2 Answers2

19

Since AR applications often run on mobile devices, you could consider also other features detector/descriptor:

Muffo
  • 1,733
  • 2
  • 19
  • 29
13

Generally if you can chose the markers you first detect a square target using an edge detector and then either Hough or simply contours - then identify the particular marker from the internal design. Rather than using a general point matcher.

Take a look at Aruco for well written example code.

Martin Beckett
  • 94,801
  • 28
  • 188
  • 263
  • 4
    Yes, the fiducial approach is the simplest, but it is not quite up-to-date. I think now one should point to textured markers. Thanks. The example is very interesting as it is well explained, though. – Jav_Rock Sep 05 '12 at 16:07
  • 1
    If you want to know the pose, you need to match the 3D position of marker features to the corresponding image coords. Regular shaped targets make this easier but they don't need to be planar – Martin Beckett Sep 05 '12 at 16:46
  • But if they are not planar, you need the 3d-model or a CAD model, otherwise homography from 2d-to-3d is not a valid method anymore, is it? I mean the Direct Linear Transform used in findhomography – Jav_Rock Sep 05 '12 at 21:53
  • Yes, if you have a 3d marker you need to know the real word 3d shape (from cad, or measuring it) but it does give a better soln – Martin Beckett Sep 05 '12 at 22:01