2

I am new to OpenCV2 and working on a project in emotion recognition and would like to align a facial image in relation to a reference facial image. I would like to get the image translation working before moving to rotation. Current idea is to run a search within a limited range on both x and y coordinates and use the sum of squared differences as error metric to select the optimal x/y parameters to align the image. I'm using the OpenCV face_cascade function to detect the face images, all images are resized to a fixed (128x128). Question: Which parameters of the Mat image do I need to modify to shift the image in a positive/negative direction on both x and y axis? I believe setImageROI is no longer supported by Mat datatypes? I have the ROIs for both faces available however I am unsure how to use them.

void alignImage(vector<Rect> faceROIstore, vector<Mat> faceIMGstore)
{
  Mat refimg = faceIMGstore[1]; //reference image
  Mat dispimg = faceIMGstore[52]; // "displaced" version of reference image
  //Rect refROI = faceROIstore[1]; //Bounding box for face in reference image
  //Rect dispROI = faceROIstore[52]; //Bounding box for face in displaced image
  Mat aligned;

  matchTemplate(dispimg, refimg, aligned, CV_TM_SQDIFF_NORMED);
  imshow("Aligned image", aligned);
}

The idea for this approach is based on Image Alignment Tutorial by Richard Szeliski Working on Windows with OpenCV 2.4. Any suggestions are much appreciated.

Abhischek
  • 189
  • 2
  • 7
  • 18
  • hi abhischek!I am having the same problem as you..could you please guide me on this http://stackoverflow.com/questions/22296058/align-profile-face-image-with-its-frontal-face-image ? – Steph Mar 16 '14 at 14:40

1 Answers1

2

cv::Mat does support ROI. (But it does not support COI - channel-of-interest.)

To apply ROI you can use operator() or special constructor:

Mat refimgROI  = faceIMGstore[1](faceROIstore[1]); //reference image ROI
Mat dispimgROI(faceIMGstore[52], faceROIstore[52]); // "displaced" version of reference image ROI

And to find the best position inside a displaced image you can utilize matchTemplate function.

Based on your comments I can suggest the following code which will find the best position of reference patch nearby the second (displaced) patch:

Mat ref = faceIMGstore[1](faceROIstore[1]);
Mat disp = faceIMGstore[52](faceROIstore[52]);

disp = disp.adjustROI(5,5,5,5); //allow 5 pixel max adjustment in any direction
if(disp.cols < ref.cols || disp.rows < ref.rows)
    return 0;
Mat map;
cv::matchTemplate( disp, ref, map, CV_TM_SQDIFF_NORMED );

Point  minLoc;
cv::minMaxLoc( map, 0, &minLoc );

Mat adjusted = disp(Rect(minLoc.x, minLoc.y, ref.cols, ref.rows));
Andrey Kamaev
  • 29,582
  • 6
  • 94
  • 88
  • Thanks for your help! I have implemented template match now, however when I output the resulting matrix it is displayed as a dot!? aligned.cols is 1 and aligned.rows is 1! It is worth mentioning that both images are the same size however are not exactly the "same" image. refimg is a neutral expression/ dispimg is a smiling expression. Do you think this will still give a match, or is this the issue causing the result matrix to be a dot? Please see updated code – Abhischek Dec 15 '12 at 16:18
  • It is expected to be the dot when images have the same size - it is in documentation. And if your images have the same size then you simply has nothing to align (or I did not get what you are trying to do). Probably you need to add some margin to the displaced image before calling matchTemplate. – Andrey Kamaev Dec 15 '12 at 16:28
  • Yes the images have the same size, however the face on the second image is slightly lower than the first, causing the eyes to be misaligned when I output the overlay using addWeighted. Hence I wanted to "push" the second image upwards to align the eye region. – Abhischek Dec 15 '12 at 16:33
  • Thanks, trying to implement the changes. In your expertise do you reckon this will work for aligning different faces as well (male/female)? – Abhischek Dec 15 '12 at 17:08
  • Hi there!interesting post.In the original function void alignImage(vector faceROIstore, vector faceIMGstore), what are meaning of the parameters? – Steph Mar 03 '14 at 16:42