10

I am developing an application for iOS that needs to detect when the user yawns.

What I did is include Open CV and find faces using a haar cascade, and then find mouth inside the faces (too using a haarcascade).

The trouble that I have is that I believed that it would be so easy to detect a yawn like doing something like (face.y - mouth.y) < something = yawn.

But the problem that I have is that the rects for face and mouth are "inestables", I mean every time that the loop runs X and Y values for the rects of face and mouth are (obviusly) not the same.

Is there any "open mouth" haar cascade that I can use, or how can I know when the user opens the mouth?

Paul Cezanne
  • 8,629
  • 7
  • 59
  • 90
iVela
  • 1,160
  • 1
  • 19
  • 40

2 Answers2

11

As a general, Support Vector Machine (SVM) is used for facial expression recognition like anger,smile, surprise etc where still active development takes place. Googling give you a lot of papers on this topic, (even one of my class mate did this as his final year project). For that, at first you need to train the SVM and to do that, you need sample images of yawning and normal faces.

Yawning is almost similar to surprise, where mouth open on both cases. I recommend you to check out page 3 of below paper : Real Time Facial Expression Recognition in Video using Support Vector Machines (If you can't access the link, google by paper name)

The paper (even my classmate) used displacement vector of facial features. For this, you find some feature points on the face. For example, in the paper, they have used eye pupil,extreme points of lids, nose tip, extreme points of mouth region (lips) etc. Then they continuously track the location of the features and find euclidean distance between them. They are used to train the SVM.

Check out below two papers :

Feature Points Extraction from Faces

Fully Automatic Facial Feature Point Detection Using Gabor Feature Based Boosted Classifiers

Look image below what i mean by feature points on face:

enter image description here

In your case, i think you are implementing it in iPhone in real time. So may be you can avoid feature points at eyes (although not an good idea, since when you yawn, eyes become small in size). But compared to it, feature points at lips shows more variations and predominant. So, implementing on lip alone may save time. (Well, it all depends on you).

Lip Segmentation : It is already discussed in SOF and check out this question : OpenCV Lip Segmentation

And finally, i am sure you can find a lot of details on googling, because it is an active development area, and a lot of papers are out there.

Another Option :

Another option in this region, which i have heard several times, is Active Appearance Model. But I don't know anything about it. Google it yourself.

Community
  • 1
  • 1
Abid Rahman K
  • 51,886
  • 31
  • 146
  • 157
  • Those concepts are far way from me :) But I've resolved in this way with the haar cascades: first I search faces, inside the faces I search the mouth. The mouth is found when the lips are not open. So if I've found a mouth and later I can't find the mouth anymore I suposse that the user has yawned. Is NOT perfect, is not even nice, but it solved what I needed to develop my application. – iVela Jun 12 '12 at 14:55
  • The method i told is not so far. This was done by my friend in matlab when we are undergrads. And also all functions are implemented in OpenCV. No problem, yours is choice. just pointed out this method. good luck – Abid Rahman K Jun 12 '12 at 15:58
  • Thnanks @Abid the trouble is that I don't have any idea about SVM and neither about Open CV, just the basics :D but it is really interesting so after this project I will research a little more to make a "good" implementation of this. And also this knowlegde have multiple uses for example augmented reality, and it can be implemented almost every where with Open CV (Android, iOS, Desktop, etc). So in the future I will research more on the methods that you told me. Thanks again! – iVela Jun 12 '12 at 18:30
  • I have been looking for the same solution. iOS face expression recognition library. Is there a sample SVM program/github to work with further on iOS? – Stella Nov 28 '16 at 18:03
0

OpenCV also has face recognition / detection capabilities (see the examples that come with the openCV SDK). I think these would be a better place to look, as haar cascade doesn't really analyze the facial expressions the way you need it to. Try running the examples and see for yourself - you'll get real time data about the detected eyes / mouth and so.

Good luck

Stavash
  • 14,244
  • 5
  • 52
  • 80