18

I am developing a C++ application for face authentication. First, I have to detect the face and pre-process the image.

  1. For face detection I have used the HaarCascadeClassifier. The problem is that the this tool or this algorithm gives me a facial region detected by a little bit large rectangle that englobes hair and some of the background. Is there a solution to change the dimension of this rectangle? I used "frontalfacecascaadclassifier.xml".
  2. For face pre-processing i want to do face alignment exactly like this technique. How would I go about accomplishing this?
Seanny123
  • 8,776
  • 13
  • 68
  • 124
OntoBLW
  • 215
  • 1
  • 2
  • 6
  • Why can't orientation detection using PCA, as shown [here](https://robospace.wordpress.com/2013/10/09/object-orientation-principal-component-analysis-opencv/) be applied to faces as well? – Seanny123 Apr 23 '14 at 03:01
  • I found [dlib](https://github.com/davisking/dlib) seems like a very powerful library, and it is fast, free and open source. – dashesy Aug 18 '15 at 18:23

9 Answers9

18

Finding the accurate position of the eyes in a given image is far from trivial. The Haar-cascades for finding the eyes in OpenCV produce too many false positive to be useful, moreover this approach won't be robust to image rotation (it may compensate slight rotation though, I don't know the training images). If I were you I'd start a breadth-first search on http://scholar.google.com for relevant papers of this research area.

You'll need a robust head pose estimation for aligning face images. I did some research myself and I think sharing algorithms and code is useful here. The most interesting approaches I have seen are:

bytefish
  • 3,697
  • 1
  • 29
  • 35
  • I want to proceed a method like this http://www.bytefish.de/blog and obtain an iamge like this http://www.bytefish.de/_detail/blog/arnie_20_20_200_200.jpg?id=blog Have an idea how to do that in c++ – OntoBLW Apr 15 '12 at 10:09
8

Can't you then use another Haar classifier to find each eye (eyes are very easy to find) then assuming the person has two eyes and we define a 'level' face to mean the eyes are horizontal.

Simply measure the anlge between the two eyes and rotate the image by that angle.

angle = atan ( eye1.Y - eye2.Y ) / (eye1.X - eye2.X )
herohuyongtao
  • 49,413
  • 29
  • 133
  • 174
Martin Beckett
  • 94,801
  • 28
  • 188
  • 263
  • thanks for your response especially your quick response.yes,i agree with you about detecting first eyes.but i haven't understood how to "define a 'level' face to mean the eyes are horizontal".Is that thanks ti that angle you defined.Really how can i have a face according to eyes position – OntoBLW Apr 13 '12 at 17:22
  • @louloulili - you said you had already found the face from a Haar, you just needed to rotate it? – Martin Beckett Apr 13 '12 at 17:57
  • i want to have an image like this http://www.bytefish.de/_media/blog/arnie_20_20_200_200.jpg?cache=&w=200&h=200 but i found an image like this http://www.bytefish.de/_media/blog/arnie_30_30_200_200.jpg and teh rectangle of haar detecting face is large.and after i will rotate it to align it.please help me and thanks in advance for your responses. – OntoBLW Apr 13 '12 at 20:23
  • 2
    @MartinBeckett Yes, that's what the Python script at http://www.bytefish.de/blog/aligning_face_images does. I think the question is about finding the position of the eyes, which isn't trivial to answer. – bytefish Apr 15 '12 at 06:10
  • @/bytefish for your Python scrip,while i am developing a c++ application,can i detect eyes position then i calculate the angle=ang= atan ( eye1.Y - eye2.Y ) / (eye1.X - eye2.X ) then do an affin transformation using this angle to have results like what is obtained in python script and exactly to have an image like this http://www.bytefish.de/_media/blog/arnie_20_20_200_200.jpg ? thanks for help – OntoBLW Apr 15 '12 at 10:22
6

I tried the following face alignment code from the Labelled Faces in the Wild project page. It works really well and does not require detecting facial feature points. The C++ code can be downloaded from: http://vis-www.cs.umass.edu/faceAlignment/

If you still wish to find face key points, I find that the Viola-Jones detector is not very robust and accurate. I personally recommend using the Flandmark face keypoint detector: http://cmp.felk.cvut.cz/~uricamic/flandmark/ which is much much more robust and accurate. C code can be downloaded from the above site.

lightalchemist
  • 10,031
  • 4
  • 47
  • 55
  • Using their `people.train` data on some of my sample images, didn't work as well as I hoped. It rotated some of my control images and didn't rotate my test images as much as I wanted. Would you mind sharing a sample of your test set and the result that it gave that gave you the confidence to make this recommendation? – Seanny123 Feb 16 '14 at 12:08
  • 1
    I'm not familiar with the people.train model that you are mentioning. I used the congealing code to train my own model which I call faces.train for faces, using about 1000 faces. Then used the funneling code to use that to align faces. Firstly, the original misalignment in your training and test images cannot be too "far" off. If they are misaligned by > 40 degrees or if the scale is too different it doesn't work. Read the paper. Don't just treat it as a black box. – lightalchemist Feb 16 '14 at 14:08
  • One of the test set I tried with is [this](http://chenlab.ece.cornell.edu/people/Andy/GallagherDataset.html). Note that how you crop the faces and the parameters, especially the inner and outer window size is very important. You want an inner window that just captures from the eyebrows down to the lips or chin. – lightalchemist Feb 16 '14 at 14:12
  • Also, don't expect it to perform miracles on faces that are facing sideways. You can try running the cars dataset they gave along with that code. That will give a sense of the size of the "windows" you need to set. – lightalchemist Feb 16 '14 at 14:13
  • Oh btw check that the people.train model/file is trained to align the type of objects you are aligning and see their test images for the "scope" of the crop i.e., what needs to be in the image. E.g., if a model file in this case is trained on faces that includes eyebrow and a bit of forehead, you should also throw in faces with eyebrows and a about the same amount of space above it for the forehead. The reason is due to how the algorithm works. – lightalchemist Feb 16 '14 at 14:20
  • (cont) It basically tries to align objects (in general, not just faces) by stacking their images together and finding (through a non-optimal greedy method) the transformation (scale, translation, and planar rotation) of the object such that the common parts are at the same position i.e., if you thread through the stack of images at the same location, you get the same part of the face. Hence, if it was trained with faces with eyebrows, it would need faces with that to help it align the face. Also, you need ard 500 decent training images to train it well. Remember, garbage in garbage out. – lightalchemist Feb 16 '14 at 14:22
  • Thanks for the elaboration. I speed read the paper, thus bringing much shame on to my family. – Seanny123 Feb 17 '14 at 00:14
  • @Seanny123 Lol. Your comment made my day. Feel free to move this to chat if you need further clarification. I did this about a year ago so the code and stuff is not easily retrievable. Also, as it was for work, it is not convenient for me to release the data and code, but I'll be happy to provide pointers if you need them. – lightalchemist Feb 18 '14 at 02:30
4

The state-of-the-art approach for face alignment must be this:

Supervised Descent Method and its Application to Face Alignment X. Xiong and F. De la Torre in CVPR 2013

It is extremely fast and effective. You can check their project website IntraFace.

They provide an easy-to-use software. However, the core part code, i.e supervised descent method (SDM) is not released, it is only simple linear regression which can be easily implemented.

A demo to show that it can handle tilted face is here (for privacy issue, add blur and pay attention to the axis in the top-left corner): https://drive.google.com/file/d/0BztytuqPViMoTG9QaEpZWi1NMGc/edit?usp=sharing

Seanny123
  • 8,776
  • 13
  • 68
  • 124
rookiepig
  • 489
  • 3
  • 12
  • Although this is a useful link, I would like more detail, for example proof that it detects tilted faces and/or the degree of tilt, before giving you the bounty. – Seanny123 Feb 19 '14 at 14:31
  • A demo is showed in the above image. For the code, what the author provided is a matlab .p file, which can be utilized in your application but the source is unknown. However, I strongly recommend you check the paper, the underlying algorithm is very simple. – rookiepig Feb 19 '14 at 14:51
  • 1
    Note that the program is actually super-easy to compile and run. However, the demo program only tracks features for a single face. – Seanny123 Apr 02 '14 at 04:19
  • @rookiepig the link to the demo is just an image...Is that on purpose or is there a demo link I can checkout? – Terence Chow Jun 03 '14 at 05:42
  • @Chowza You can check the paper's project page. [IntraFace](http://www.humansensing.cs.cmu.edu/intraface/index.php) – rookiepig Jun 03 '14 at 07:23
  • the download links for that project seem to be missing (or maybe it is the part you said is not released?) – dashesy Aug 18 '15 at 01:14
  • 1
    I found a true state-of-the-art method that is also FOSS and patent-free [dlib](https://github.com/davisking/dlib). – dashesy Aug 18 '15 at 18:20
1

Detecting misaligned faces make face recognition difficult. Sometimes you want to fix the alignment, sometimes it is sufficient to exclude the ones that aren't aligned correctly (for instance if you're detecting faces in a video stream). I took the latter approach and trained a special Haar Cascade to only detect correctly aligned, well-lit faces. Details here: http://rwoodley.org/?p=417.

If you use my cascade let me know how it works for you. I'm curious what results others would obtain. It met my needs.

Bob Woodley
  • 1,246
  • 15
  • 30
1

I have implemented it here using OpenCV & DLib: https://github.com/ManuBN786/Face-Alignment-using-Dlib-OpenCV

Any tilted faces can be aligned using my code.

Manu B.N
  • 75
  • 1
  • 5
0

Have a look at CSIRO face analysis SDK (website and demos, source code) software, it does face alignment, tracking with 66 fiducial points. It is fast and very accurate.

Seanny123
  • 8,776
  • 13
  • 68
  • 124
ivan_a
  • 613
  • 5
  • 12
  • I am using it for my thesis. I can say that it is one of the best face alignment software that I have seen. It tracks 66 landmarks in real-time. Have a look at this paper for more details "Deformable model fitting by regularized mean- shifts". – ivan_a Feb 16 '14 at 07:33
  • This program did not work on images with a tilted face. – Seanny123 Feb 16 '14 at 11:04
  • It uses OpenCV face detector for detecting faces. You can try initialize CSIRO tracker with some other face detector that works with tilted faces. However, usually for the authentication problems you will need to narrow down set of variables to a minimum, making every trained images to be as similar to the test ones as possible. – ivan_a Feb 16 '14 at 18:01
0

After searching all day for an algorithm to accomplish this with, I found "Face Detection By Finding The Facial Features And The Angle of Inclination of Tilted Face " by Hemlata et al. after switching from Google to DuckDuckGo. It supports faces that are tilted at an angle larger than 45 degrees.

As for how to implement in code, that's another problem that I'm currently working on, but at least this is a starting point.

Seanny123
  • 8,776
  • 13
  • 68
  • 124
0

For face authentication, you can use dlib or face_recognition to do this which is very convenient and more accurate than opencv now.

As to dlib, face alignment can be found here (C++ code) http://dlib.net/face_alignment.py.html

or here(python code)https://www.pyimagesearch.com/2017/05/22/face-alignment-with-opencv-and-python/.

The algorithm paper named Face Alignment at 3000 FPS via Regressing Local Binary Features is realized by dlib.

Jayhello
  • 5,931
  • 3
  • 49
  • 56