Segment a region in Depth Image of Microsoft Kinect using Open CV Python

Question

I am working on Hand Segmentation using Microsoft Kinect Depth Stream. By doing some thresholding I was able to separate body from background, based on depth information provided by MS Kinect Sensor now I want is to segment hands. Any suggestions how can I segment Hands. Below is a screen shot of my depth Image. I am using Open CV in Python. To access Kinect Sensor streams, I am using PyKinect from Microsoft.

Answer 1

What you need is a semantic segmentation algorithm. Random decision forests gave fantastic results. Check the work published by Haggag et al. It is built on top of the work done by Shotton et al and Buys et al.

If you want to use deep learning, check the work done by Abubakr et al.The base papers are published by Shotton et al. and Buys et al. Shoton's work is the one that founded Kinect.

[1] Shotton et al., "Efficient human pose estimation from single depth images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013.

[2] Buys et al., "An adaptable system for RGB-D based human body detection and pose estimation,” Journal of Visual Communication and Image Representation, 2014.

Haggag et al. and Abobakr et al. built on these papers to produce the following using random forests.

[3] Abobakr et al., "A Skeleton-Free Fall Detection System from Depth Images Using Random Decision Forest," IEEE Systems Journal, 2018.

[4] Haggag et al., "An adaptable system for RGB-D based human body detection and pose estimation: Incorporating attached props," IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2016.

[5] Haggag et al., "Semantic body parts segmentation for quadrupedal animals," IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2016.

For better voting scheme and nominal label/decision fusion of the label maps produced by the random trees, check my paper below.

[6] Hossny et al., "Decision Fusion of Labelled Images with Local Majority Filter,” Electronics Letters, 2017.

Abobakr et al., have then incorporated deep learning.

[7] Abobakr et al., "Body joints regression using deep convolutional neural networks," IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2016.

[8] Abobakr et al., "RGB-D human posture analysis for ergonomie studies using deep convolutional neural network," IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2017.

[9] Nahavandi et al., "A skeleton-free body surface area estimation from depth images using deep neural networks," IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2017.

Answer 2

What you need is a semantic segmentation algorithm. Random decision forests gave fantastic results. Check the work published by Haggag et al. It is built on top of the work done by Shotton et al and Buys et al.

If you want to use deep learning, check the work done by Abubakr et al.