How to detect the line of sight of a person using kinect?

Question

Currently I am working on a project using Kinect which requires me to know the where the person is looking at that time, for which I figured out I need to find the line of sight of that person.

Right now, I can find the head point of the skeleton of the person but can't track the eye movement.

if (body.TrackingState == SkeletonTrackingState.Tracked)
{
    Joint joint = body.Joints[JointType.Head];
    SkeletonPoint skeletonPoint = joint.Position;

    // 2D coordinates in pixels
    System.Drawing.Point point = new System.Drawing.Point();

    if (_mode == CameraMode.Color)
    {
        // Skeleton-to-Color mapping
        ColorImagePoint colorPoint = _sensor.CoordinateMapper.MapSkeletonPointToColorPoint(skeletonPoint, ColorImageFormat.RgbResolution640x480Fps30);

        point.X = colorPoint.X;
        point.Y = colorPoint.Y;
        //Console.WriteLine(" X == " + point.X + " Y == " + point.Y);
        X = (int)Math.Floor(point.X + 0.5);
        Y = (int)Math.Floor(point.Y + 0.5);
    }

    // DRAWING...
    Ellipse ellipse = new Ellipse
    {
        Fill = System.Windows.Media.Brushes.LightBlue,
        Width = 20,
        Height = 20
    };

     Canvas.SetLeft(ellipse, point.X - ellipse.Width / 2);
     Canvas.SetTop(ellipse, point.Y - ellipse.Height / 2);

     canvas.Children.Add(ellipse);
 }

Here point.X and point.Y are the head points of the skeleton.

I haven't messed with the Kinect SDK in forever but isn't it possible to detect the rotation of the head? — John Odom, May 28 '15 at 19:21
@JohnOdom how do you think to use rotation of the head in my problem....cant get your idea..though i can figure out changes in head position using skeleton coordinates — Vipul Jain, May 28 '15 at 19:50
I was thinking of if you were able to keep track of the head's rotation (0 degree for when facing the kinect) you can figure out the "line of sight" by checking how much it rotated. Ex: If the head rotated +45 degrees on the yaw (up/down Z axis) then the user is looking to the left. It may sound like overkill but it's all I can offer without looking into the kinect sdk. — John Odom, May 28 '15 at 19:53
@JohnOdom 2 things 1. Rotation of the person around the yaw will not change his head points 2. your method neglects the movement of the eye without the movement of head. — Vipul Jain, May 28 '15 at 19:58
I know I'm neglecting the eye movement, I wasn't sure if the kinect was advance enough to read eye movements. For the rotation I meant the head's rotation, not the body. I also wasn't sure if the Kinect could keep track of the body part's rotation individually. — John Odom, May 28 '15 at 20:08
Not a Kinect developer, but based on [what I could find](https://www.microsoft.com/en-us/kinectforwindows/MeetKinect/FAQ.aspx), 2.0 has facial tracking (well, they've had it since 1.5 it appears), but as far as eyes are concerned, the extent is "open or closed". (_See "Developer" section, "Does Kinect windows Windows SDK offer facial recognition capabilities?" sub-section_) — Brad Christie, May 28 '15 at 20:48
See if [this SO question](http://stackoverflow.com/questions/17155573/how-to-track-eyes-using-kinect-sdk) helps. — Icemanind, May 28 '15 at 21:34

score 1 · Accepted Answer · edited Jun 20 '20 at 09:12

Have you looked at the FaceBasics Sample Project?

I believe you want to use the FaceFrameSource / FaceFrameReader (note: not HDFace). You'll be able to get the face orientation as a Quarternion (and the sample project translates that to Pitch/Yaw/Roll).

Combining that with the 3D location of the head from the skeleton, I think you should be able to create an approximate line of sight.

The How-to Videos cover Face including some information on Orientation (5th video, skip in about 18:20 - your specific question is asked at 21:49).

EDIT: Rough proof of concept showing modifications made to FaceBasics Sample Project - added to ~line 565, right after the face info is drawn (I also needed to change the scope of pitch/yaw/roll defined a few lines above and set their default values to 0). This creates a circle for a head, and a yellow line looking at an approximate gaze location.

Joint HeadJoint = this.bodies[faceIndex].Joints[JointType.Head];
ColorSpacePoint colorPoint = this.coordinateMapper.MapCameraPointToColorSpace(HeadJoint.Position);
Point HeadPoint = new Point(colorPoint.X, colorPoint.Y);
Point GazePoint = new Point(HeadPoint.X - Math.Sin((double)yaw * 0.0175) * 600, HeadPoint.Y - Math.Sin((double)pitch * 0.0175) * 600);
drawingContext.DrawLine(new Pen(System.Windows.Media.Brushes.Yellow, 5), HeadPoint, GazePoint);
drawingContext.DrawEllipse(System.Windows.Media.Brushes.LightBlue, null, HeadPoint, 70, 70);

EDIT 2: Just saw your new comment saying that you are using SDK v1.8 - my answer is going off of v2.0, and I can't speak for how things would be different with the older SDK/Sensor.

yes I have seen those videos, and modifying the facebasics code for line of sight seem to me a very difficult task as it contained different modules which were interdependent on each other...thanks — Vipul Jain, May 29 '15 at 09:56
I think you're going to have a very hard time if you want to track face orientation but without using the FaceFrameSource. — GregT-MN, Jun 01 '15 at 18:25
just got the kinect v2 and adapter...will be implementing your suggestions now...will let you know about further updates — Vipul Jain, Jun 07 '15 at 19:12

How to detect the line of sight of a person using kinect?

1 Answers1