0

I am using Object segmentation dataset having following information:

Introduced: IROS 2012

Device: Kinect v1

Description: 111 RGBD images of stacked and occluding objects on table.

Labelling: Per-pixel segmentation into objects.

link for the page: http://www.acin.tuwien.ac.at/?id=289

I am trying to use the depth map provided by the dataset. However, it seems the depth map is completely black.enter image description here

Original image for the above depth map enter image description here

I tried to do some preprocessing and normalised the image so that the depth map could be visualised in the form of a gray image.

img_depth = cv2.imread("depth_map.png",-1) #depth_map.png has uint16 data type
depth_array = np.array(img_depth, dtype=np.float32)
frame = cv2.normalize(depth_array, depth_array, 0, 1, cv2.NORM_MINMAX)
cv2.imwrite('capture_depth.png',frame*255)

The result of doing this preprocessing is: enter image description here

In one of the posts in stackoverflow, i read that these black patches are the regions where the depth map was not defined.

If i have to use this depth map, what is the best possible way to fill these undefined regions? (I am thinking of filling these regions with K-nearest neighbour but feel there could be better ways for this).

Are there any RGB-D datasets that do not have such problems or these kind of problems always exists? what are the best possible way to tackle such problems?

Thanks in Advance!

kkk
  • 1,850
  • 1
  • 25
  • 45
  • before training you should make sure your images are aligned see [Align already captured rgb and depth images](https://stackoverflow.com/a/35914008/2521214) also by enhancing dynamic range of depth channel you lose units in depth ... that is not a good idea because then you can not distinct between thin and wide objects – Spektre Jul 28 '17 at 06:08

2 Answers2

1

Pretty much every 3d imaging technology will produce data with invalid or missing points. Lack of texture, too steep slopes, obscuration, transparency, reflections,... you name it.

There is no magic solution to filling these holes. You'll need some sort of interpolation or you maybe replace missing points based on some model.

The internet is full of methods for filling holes. Most techniques for intensity images can be successsfully applied to depth images.

It will depend on your application, your requirements and what you know about your objects.

Data quality in 3d is a question of time, money and the right combination of object and technology.

Piglet
  • 27,501
  • 3
  • 20
  • 43
  • thanks for your answer, but can you help me suggesting some methods that best does this. I am new to this field. My problem is huge and one of the problems that i am facing is getting the correct depth map. So how best to fit this depth map so minimal loss to the image is done. – kkk Jul 27 '17 at 15:54
  • 2
    @kkk I can't give you a "best" method because I don't know what you mean with best. I hate the word "best". Everyone wants the best but no one specifies what this is supposed to mean. Ask 50 guys what the best pizza is. I bet you get 40 different answers and 20 of them you don't like.... I don't even know what you want to achieve. In most applications these holes don't matter at all or even cause problems if filled. And if you are new to this field, don't start with the "best", start with something that is simple then proceed to more advanced things nearest neigbour, bilinear, bicubic,... – Piglet Jul 27 '17 at 16:04
  • I have to use this depth maps to train neural networks so that given any image i can get the depth map of it. As you would know, dataset is an important part in training these kind of networks, having a dataset that have holes could lead to results that are unwanted and could mislead many of my results. – kkk Jul 27 '17 at 16:57
  • @kkk sorry but you're missing what? like 2% of your object. and to be honest you also miss the entire back and bottom half of it. come on. also filling these holes will only create artificial that might look better but won't help you. it might as well hurt your training more than it helps. training with false data is worse than training with less but correct data. you will always have holes in depth images. you should be more than happy that you can get such data at all with a sensor you basically get for free. – Piglet Jul 27 '17 at 17:06
  • 64000 pixels values are undefined in my case out of 307000 pixels in the image, that constitute to about 20% of the image and a big concern in my case. I am asking for suggestions if someone would have faced a similar problem and could have came with an elegant solution to it. – kkk Jul 27 '17 at 17:13
  • 1
    @kkk I was talking about the missing stuff at the boxes. who cares about the rest? how do you want to label that? defocussed strip of wall? restrict your images to the center region. the rest has terrible quality anyway. and as I said, just exclude those pixels from learning. they are not part of your data! – Piglet Jul 27 '17 at 17:26
  • Thanks a lot, i will try to ignore the values that are there near the boundaries. – kkk Jul 27 '17 at 17:43
1

Areas that absorb or scatter the Kinect IR (like glossy surfaces or sharp edges) are filled with zero pixel value (indicating non-calculated depth). A method to approximately fill the non-captured data around these areas is by using the statistical median of a 5x5 window. This method works just fine for Kinect depth images. An example implementation can be seen for Matlab and C# in the links.

Atif Anwer
  • 450
  • 5
  • 13