Kinect v2, mapping from color frame to camera space

Question

I have one pixel in 1920*1080 color frame, and I need to know it's location in camera space in meters. I know I should use CoordinateMapper class, but the method CoordinateMapper.MapColorFrameToCameraSpace documented here takes depth frame as input. I'm confused: shouldn't the input be a color frame? I want to map between color frame and camera space after all.

I think there's something eludes me, I appreciate it if anyone can make it clear. Thank you!

GregT-MN · Answer 1 · 2015-06-05T03:19:11.533

0

This is more a comment than an answer (but I don't have the rep to comment):

I believe the reason it requires a depth frame and not just a color frame is that camera space is three-dimensional, so it couldn't know that just from a 2D pixel location - it needs the depth.

edited Jun 05 '15 at 03:19

answered Jun 05 '15 at 01:19

GregT-MN

164
1
10

But it still needs the color frame along with depth frame, right? I don't know how to send color frame in. – chunjiw Jun 05 '15 at 01:25
[It looks like](https://social.msdn.microsoft.com/Forums/en-US/554ac054-3b15-4a36-a7ff-4507ff393258/coordinatemapper-mapcolorframetodepthspace-and-mapcolorframetocameraspace-methods?forum=kinectv2sdk) the method creates a lookup table so that you can get the color value at a given depth location. – GregT-MN Jun 05 '15 at 02:02
Looking at the code highlighted in that link (and the CoordinateMapper-Basics sample code that came with the SDK) it iterates through the entire 1920x1080 pixels in the color space and checks the values at that point. So, if I'm understanding [this](http://stackoverflow.com/questions/20181132/edit-raw-pixel-data-of-writeablebitmap) correctly, the depth info for the pixel (200,100) on your color image could be looked at by skipping straight to colorIndex = (1920 x 100 + 200) in that for loop from the sample project. – GregT-MN Jun 05 '15 at 02:26

score 0 · Answer 2 · answered Jun 05 '15 at 01:29

Check this out... This code is something I built for Halloween. It demonstrates (sort of) what you're looking for. The comments in the code help too.

http://github.com/IntStarFoo/KinectAStare

http://github.com/IntStarFoo/KinectAStare/blob/master/ViewModels/KinectBaseViewModel.cs

                    TrackedHead = body.Joints[JointType.Head].Position;
                    //This is an 'aproxometery'  http://trailerpark.wikia.com/wiki/Rickyisms
                    //  of the tracking direction to be applied to the eyeballs on 
                    //  the screen.
                    TrackedHeadX = (int)(TrackedHead.X * 10);
                    TrackedHeadY = (int)(TrackedHead.Y * -10);

                    // Really, one should map the CameraSpacePoint to 
                    //  the angle between the location of the eyes on 
                    //  the physical screen and the tracked point. And stuff.                        //This is the TrackedHead Position (in Meters)
                    //The origin (x=0, y=0, z=0) is located at the center of the IR sensor on Kinect
                    //X grows to the sensor’s left
                    //Y grows up (note that this direction is based on the sensor’s tilt)
                    //Z grows out in the direction the sensor is facing
                    //1 unit = 1 meter

                    //Body
                    //body.Joints[JointType.Head].Position.X;
                    //body.Joints[JointType.Head].Position.Y;
                    //body.Joints[JointType.Head].Position.Z;

                    //Kinect (0,0,0)

                    //Screen Eyes (?,?,?)

Thank you for your answer, but in your code I only found mapping between depth and color frame. I didn't see what you did to the 3D CameraSpacePoint. Maybe it's there but I didn't find it (I'm only a beginner when it comes to programming). Also, are we trying to do different things here? Because I'm trying to map color frame to camera space, not the other way around. But your code inspires me, maybe I can map color frame to depth frame, and then map depth frame to camera space. I'm wondering why this is not already provided. — chunjiw, Jun 08 '15 at 18:01

José Ernesto Lara Rodríguez · Accepted Answer · 2015-06-08T14:04:39.803

0

The reason it doesn't ask the color frame is because it doesn't need it. This method maps every possible pixel in a color frame to its corresponding 3D coordinate. For that, it needs the depth frame, which is the one that contains 3D depth information, which allows the software to know where in 3D space each of the points of that 2D image would be (I don't know how they do it, but I imagine it can be done with raycasting). If you think about it, there is no way of reconstructing the 3D world from a simple image (which only contains color information in each point). If there was, there would be no need for Kinect at all, right? We could get depth information from simple cameras :)

Hope my answer helped you understand, if something isn't clear, feel free to ask.

edited Jun 08 '15 at 14:04

answered Jun 05 '15 at 20:49

José Ernesto Lara Rodríguez

1,347
1
13
26

Thank you for your answer, I understand that it needs depth frame, but I think it needs color frame as well. What I want is a mapping between "one pixel in color frame" and "one point in camera space", if I cannot pass in the former, how do I get the latter? – chunjiw Jun 08 '15 at 16:59
That method you linked maps all the pixels from the color frame to camera space, not just a single pixel. That's why it doesn't lets you pass the pixel you want. It fills the array `cameraSpacePoints` with all the mapped pixels. After that, you can get the pixel you want using `cameraSpacePoints[y * colorImageWidth + x]`, where `x` and `y` are the coordinates of the pixel you want. – José Ernesto Lara Rodríguez Jun 08 '15 at 17:15
Thank you very much, now all my confusion is lifted. – chunjiw Jun 08 '15 at 20:56
Glad to be helpful :) – José Ernesto Lara Rodríguez Jun 08 '15 at 22:06

Kinect v2, mapping from color frame to camera space

3 Answers3