4

WE are doing a project using kinect and opencv. I am completely new to 3D analysis. Main aim is to take the depth and rgb information from kinect,process the rgb mat ( basic filtering and threshold functions) and combine the processed rgb image with the original depth information. i need basic steps to follow with suitable applications to use ( opencv,opengl,openni,kinect sdk etc)

1) how to pass the depth and rgb information from kinect to opencv?

2) can we individually access the rgb image to process it?

3) how to combine the two and what functions to use to display the output?

We are using windows-64 bit and kinect as 3D sensor. We are planning to use OpenNI to get the image rgb and depth information and then process rgb in OpenCV and then display the Image (Processed RGB + depth) in a window with help of OpenGL.

Ashok Varma
  • 3,489
  • 3
  • 28
  • 43
Anudeep Varma
  • 78
  • 1
  • 6

1 Answers1

3

First of all, I think your question is too general, it will depend on what libraries/drivers you use... For instance, if you use openni then you pass the information one way and if you use the kinect sdk you will have to use another method...

2) yes you can access the RGB images independently of the depth information and process it...

3) to combine the two, I suppose you meant depth and rgb. This is also dependent on the libraries you use and the drivers. Also the way you want to store it, e.g. point clouds, image, etc.

My suggestion is to define exactly what you want to achieve, with which libraries and in which operating system, and edit your question.

I created a tool using openni2, opencv and Qt, I have only tested with primesense cameras and structure.io cameras in linux, but you may get an idea of what to do exactly. This tool can do some basic thresholding and saving the data in different formats (pcd, images, oni). Link

I also used almost the same approach with openni1 and Avin's driver and a Kinect1 camera, on wednesday I can upload the code and you can take a look, but basically is the same, only the initialization changes a little bit.

As a basic checklist of things you should do when working with RGB-D images:

  • Initialize the camera streams
  • Create a matrix in opencv and copy the data directly to it, be sure to read the standard of the library, for example, openni uses RGB arrays for color matrix, while KinectSDK uses RGBA (at least kinect SDK 2.0). Also make sure that the depth matrix is 16 bits.
  • To display the depth matrix you must normalize it and change it to a greyscale matrix.
  • You may do any manipulation to the images and then save them in the desire format.
  • Check the link I gave you to see how to save it in the most common formats.

Also you may take a look at the Point Cloud Library, they have a nice wrapper that will do the job and then it is easy to turn it into a pointcloud.

I hope this information helps you.

UPDATE: For a Kinect 1 in windows, it is possible to install the latest Kinect SDK and use openni2 without installing additional drivers.

  • just install Kinect SDK 1.8 (not the 2.0)
  • Install Openni2
  • run NiViewer.exe to test it works (it is in the openni folder)

If everything is good up to here, then you only need to:

  • Initialize openni
  • Initialize color and depth stream
  • do a main loop where you will copy it to a OpenCV mat (you may use my code to see an example of what I mean)
  • do any process you want to the rgb image
  • create a point cloud for 3D display/manipulation, for this I recommend Point Cloud Library, since it is easy to create a point cloud and they have already a display for it.

I can give you code snippets if you are in doubt in any of the steps, just ask a new question about it and I will answer it.

I hope this helps you

api55
  • 11,070
  • 4
  • 41
  • 57
  • We added the details to question Can you help with code ?? Code to get the rgb Mat to openCV and finally display the 3d Image. We need to just capture 3D image and do some thresholding and image processing in openCV and finally display the image in a 3D view. We completed the OpenCV code. so now we need your help to capture an image and we need RGB mat for opencv processing and then finally display that processed rgb mat with depth in a 3D view. – Ashok Varma Mar 23 '15 at 02:57
  • @AshokVarma is it Kinect or Kinect2? – api55 Mar 23 '15 at 05:23
  • it is kinect not kinect2 – Ashok Varma Mar 23 '15 at 05:26
  • ThankYou verymuch. and one final doubt, can we directly convert OpenCV mat to PointCloud ?? and is pointColud library better than OpenGL for 3D images – Ashok Varma Mar 23 '15 at 08:28
  • 1
    @AshokVarma You can convert it to point cloud, you just need to go through the depth image pixels and in the same place in the rgb image you will find the corresponding color, the z value will be the depth image pixel, and the x and y values you will need to calculate it something [like this](http://vision.in.tum.de/data/datasets/rgbd-dataset/intrinsic_calibration) pcl have already several algorithms for the point clouds... in opengl you will have to create them yourself, but you can do more things – api55 Mar 23 '15 at 14:59
  • here is the new question (http://stackoverflow.com/questions/29270544/how-to-display-a-3d-image-when-we-have-depth-and-rgb-mats-in-opencv-captured-f) thankyou. OpenNI worked fine, we need some help with Display – Ashok Varma Mar 26 '15 at 03:25