0

I am getting confused with the Kinect v2 CoordinateMapper API. I want to get RGB values of each 3D point aka CameraSpacePoint in Kinect v2.

Please see below the code snippet:

var depthFrame = multiSourceFrame.DepthFrameReference.AcquireFrame();
var colorFrame = multiSourceFrame.ColorFrameReference.AcquireFrame();

var depthWidth = depthFrame.FrameDescription.Width;
var depthHeight = depthFrame.FrameDescription.Height;

ushort[] depthData = new ushort[depthWidth * depthHeight];
CameraSpacePoint[] camerapoints = new CameraSpacePoint[depthData.Length];
ColorSpacePoint[] colorpoints = new ColorSpacePoint[depthData.Length];

depthFrame.CopyFrameDataToArray(depthData);
this.coordinateMapper.MapDepthFrameToCameraSpace(depthData, camerapoints);
this.coordinateMapper.MapDepthFrameToColorSpace(depthData, colorpoints);

The 3D points are stored in camerapoints variable. I want to the RGB values of each camerapoints. In other words, please see below pesudo code:

RGBPoint[] rgbpoints = new RGBPoint[depthData.Length];
RGBPoint rgbpoint = rgbpoints[0];
int red   = rgbpoint.r
int green = rgbpoint.g
int blue  = rgbpoint.b

As always, thank you very much. I really appreciate your kind response.

ravi
  • 6,140
  • 18
  • 77
  • 154

1 Answers1

1

The colorpoints array contains the (x, y) index that corresponds to each pixels in the depthData array. From here, we can just get the color frame data (from colorFrame) at each specific (x, y) index indicated by colorpoints[index].

var colorWidth = colorFrame.FrameDescription.Width;
var colorHeight = colorFrame.FrameDescription.Height;

// Assuming BGRA format here
byte[] pixels = new byte[colorWidth * colorHeight * 4];
colorFrame.CopyFrameDataToArray(pixels);

byte[] bgraPoints = new byte[depthWidth * depthHeight * 4];

for (var index = 0; index < depthData.Length; index++)
{
    var u = colorpoints[index].X;
    var v = colorpoints[index].Y;
    if (u < 0 || u >= colorWidth || v < 0 || v >= colorHeight) continue;
    var pixelsBaseIndex = v * colorWidth + u;
    Array.Copy(pixels, 4 * pixelsBaseIndex, bgraPoints, 4 * index, 4);
}

var index = 0;
var red = bgraPoints[4 * index + 2];
var green = bgraPoints[4 * index + 1];
var blue = bgraPoints[4 * index];
Tuwuh S
  • 291
  • 1
  • 7
  • Thank you very much. But it is throwing following error `ArgumentOutOfRangeException: Number was less than the array's lower bound in the first dimension.` I have added a separate question and mentioned more info about it. Please see [here](https://stackoverflow.com/q/47451730/1175065) – ravi Nov 23 '17 at 12:22
  • Interesting, actually I haven't really tried executing the code (don't have a Kinect on hand right now). However my guess this is because some depth pixels might be outside the frame of color pixels, due to offset and field of view between the two cameras. One way to handle this is just to ignore those values. Updated my answer. – Tuwuh S Nov 25 '17 at 18:04
  • Thank you very much. I will try it tomorrow. One more question `pixelsBaseIndex = v * depthWidth + u` instead of `depthWidth` should it be `colorWidth` ? Not sure though since I am out of laboratory now. Thanks again for your updated answer. – ravi Nov 26 '17 at 01:26
  • Something is still fishy. I couldn't get it done. Can you please check out [here](https://stackoverflow.com/q/47826550/1175065) – ravi Dec 16 '17 at 14:58