Get coordinates of non-geometric lines in a binary image

Question

I'm trying to recognize hand positions in OpenCV for Android. I'd like to reduce a detected hand shape to a set of simple lines (= point sequences). I'm using a thinning algorithm to find the skeleton lines of detected hand shapes. Here's an exemplary result (image of my left hand):

In this image I'd like to get the coordinates of the skeleton lines, i.e. "vectorize" the image. I've tried HoughLinesP but this only produces huge sets of very short lines, which is not what I want.

My second approach uses findContours:

// Get contours
Mat skeletonFrame; //image above
ArrayList<MatOfPoint> contours = new ArrayList<MatOfPoint>();
Imgproc.findContours(skeletonFrame, contours, new Mat(), Imgproc.RETR_CCOMP, Imgproc.CHAIN_APPROX_SIMPLE);

// Find longest contour
double maxLen = 0;
MatOfPoint max = null;
for (MatOfPoint c : contours) {
    double len = Imgproc.arcLength(Util.convert(c), true);  //Util.convert converts between MatOfPoint and MatOfPoint2f
    if (len > maxLen) {
        maxLen = len;
        max = c;
    }
}

// Simplify detected contour
MatOfPoint2f result = new MatOfPoint2f();
Imgproc.approxPolyDP(Util.convert(max), result, 5.0, false);

This basically works; however, the contours returned by findContours are always closed, which means that all the skeleton lines are represented twice.

Exemplary result: (gray lines = detected contours, not skeleton lines of first image)

So my question is: How can I avoid these closed contours and only get a collection of "single stroke" point sequences?

Did I miss something in the OpenCV docs? I'm not necessarily asking for code, a hint for an algorithm I could implement myself would also be great. Thanks!

Answer 1

I would start with real hand skeleton as kinematic

1. find fingers endpoints and hand/wrist base and perimeter boundary (red)

2. solve inverse kinematics

   for example by CCD to match the fingers endpoints and not overlapping image. This way you should obtain anatomically correct answer

3. for simplification you can use kinematics like this

   

   you should handle male/female/child differently (different finger lengths) or use some kind of calibration or measurement because of the different finger lengths. As you can see I skip the hand/wrist base bones they are not that important. The red outline can be found where perimeter has smaller curve radius.

How to solve your problem in your current implementation?

The first thinning approach is better so when you got the huge set of lines connect them to polylines after that compute angle of each line. If two joined lines have similar angle (up to treshold) then join them that should do what you want but do not expect you will get the lines similar to human bones especially for curves the result will be much different. In both count of lines and shape.

For better result you need to use geometrical thinning

But I have no Idea if it is present in OpenCV (I do not use this lib) the idea is to find the perimeter line and shift it perpendicular inwards by some small step similar to this. Stop if desired width is acquired

When shifted perimeter leads to too thin shape stop there and connect to thinned point from previous step (yellow line). This is all done on vectors (polylines) not on image pixels !!! Width can be computed as smallest perpendicular distance to any nearby line.