Im currently working on a robot application, where the robot should be "controlled" by the Azure Kinect. Goal is to take a picture of the scene, then I want the robot to move to the position that I clicked in the picture. Therefore i have to transform the image coordinates (u,v) to the robots base frame (x,y,z). The camera was calibrated using OpenCV and Python, so that the camera matrix and the distortion coefficient are available. The camera will be static and won't move. The pixel coordinates are gained from the transformed color depth image (i.e. the color image is transformed to the geometry of the depth camera). The calibration was also done with that images.
With given Depth-Map I was following this approach and calculated the coordinates:
z = depth_image[int(v), int(u)] #get depth value from u,v coordinates
x = z / fx * (u - cx)
y = z / fy * (v - cy)
The calculated values are w.r.t. the camera coordinate system. I marked the cx and cy value in the image with OpenCV and w.r.t. that i measured the distance from that point to the origin of my world coordinate system. I added these distances as an offset to the previous calculated x and y value. (Is that right?)
Another approach was following this topic. It worked pretty well but only when the Z-Values of the objectPoints were really close together or were the same. There are deviations from 10 up to 30mm.
Following this tutorial i wasn't able to use it with the depth camera but it worked quite good with the color camera for x and y values, not for the z values.
Are there other methods to solve the problem or am I on the right track using these methods? Maybe there is a way to multiply the u,v coordinates with a rotation and translation vector/matrices?
Should I use the transofrmed color to depth image as now, or it is bette to use only depth or color image?
