How to segment bent rod for angle calculations?

Question

I'm trying to use OpenCV to segment a bent rod from it's background then find the bends in it and calculate the angle between each bend.

The first part luckily is trivial with a enough contrast between the foreground and background. A bit of erosion/dilation takes care of reflections/highlights when segmenting.

The second part is where I'm not sure how to approach it.

I can easily retrieve a contour (top and bottom are very similar so either would do), but I can't seem to figure out is how to get split the contour into the straight parts and the bend rods to calculate the angles.

So far I've tried simplyfying the contours, but either I get too many or too few points and it feels difficult to pin point the right settings to keep the straight parts straight and the bent parts simplified.

Here is my input image(bend.png)

And here's what I've tried so far:

#!/usr/bin/env python
import numpy as np
import cv2

threshold = 229
# erosion/dilation kernel
kernel = np.ones((5,5),np.uint8)
# contour simplification
epsilon = 0

# slider callbacks
def onThreshold(x):
    global threshold
    print "threshold = ",x
    threshold = x
def onEpsilon(x):
    global epsilon
    epsilon = x * 0.01
    print "epsilon = ",epsilon

# make a window to add sliders/preview to
cv2.namedWindow('processed')
#make some sliders
cv2.createTrackbar('threshold','processed',60,255,onThreshold)
cv2.createTrackbar('epsilon','processed',1,1000,onEpsilon)
# load image
img = cv2.imread('bend.png',0)
# continuously process for quick feedback
while 1:
    # exit on ESC key
    k = cv2.waitKey(1) & 0xFF
    if k == 27:
        break

    # Threshold
    ret,processed = cv2.threshold(img,threshold,255,0)
    # Invert
    processed = (255-processed)
    # Dilate
    processed = cv2.dilate(processed,kernel)
    processed = cv2.erode(processed,kernel)
    # Canny
    processed = cv2.Canny(processed,100,200)

    contours, hierarchy = cv2.findContours(processed,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
    if len(contours) > 0:
        approx = cv2.approxPolyDP(contours[0],epsilon,True)
        # print len(approx)
        cv2.drawContours(processed, [approx], -1, (255,255,255), 3)
        demo = img.copy()
        cv2.drawContours(demo, [approx], -1, (192,0,0), 3)
    # show result
    cv2.imshow('processed ',processed)
    cv2.imshow('demo ',demo)


# exit
cv2.destroyAllWindows()

Here's what I've got so far, but I'm not convinced this is the best approach:

I've tried to figure this out visually and what I've aimed for is something along these lines:

Because the end goal is to calculate the angle between bent parts something like this feels simpler:

My assumption that fitting lines and compute the angles between pairs of intersecting lines could work:

I did a quick test using the HoughLines OpenCV Python tutorial, but regardless of the parameters passed I didn't get great results:

#!/usr/bin/env python
import numpy as np
import cv2

threshold = 229
minLineLength = 30
maxLineGap = 10
houghThresh = 15

# erosion/dilation kernel
kernel = np.ones((5,5),np.uint8)

# slider callbacks
def onMinLineLength(x):
    global minLineLength
    minLineLength = x
    print "minLineLength = ",x

def onMaxLineGap(x):
    global maxLineGap
    maxLineGap = x
    print "maxLineGap = ",x

def onHoughThresh(x):
    global houghThresh
    houghThresh = x
    print "houghThresh = ",x

# make a window to add sliders/preview to
cv2.namedWindow('processed')
#make some sliders
cv2.createTrackbar('minLineLength','processed',1,50,onMinLineLength)
cv2.createTrackbar('maxLineGap','processed',5,30,onMaxLineGap)
cv2.createTrackbar('houghThresh','processed',15,50,onHoughThresh)
# load image
img = cv2.imread('bend.png',0)
# continuously process for quick feedback
while 1:
    # exit on ESC key
    k = cv2.waitKey(1) & 0xFF
    if k == 27:
        break

    # Threshold
    ret,processed = cv2.threshold(img,threshold,255,0)
    # Invert
    processed = (255-processed)
    # Dilate
    processed = cv2.dilate(processed,kernel)
    processed = cv2.erode(processed,kernel)
    # Canny
    processed = cv2.Canny(processed,100,200)

    lineBottom = np.zeros(img.shape,np.uint8)

    contours, hierarchy = cv2.findContours(processed,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
    if len(contours) > 0:
        cv2.drawContours(lineBottom, contours, 0, (255,255,255), 1)

    # HoughLinesP
    houghResult = img.copy()
    lines = cv2.HoughLinesP(lineBottom,1,np.pi/180,houghThresh,minLineLength,maxLineGap)
    try:
        for x in range(0, len(lines)):
            for x1,y1,x2,y2 in lines[x]:
                cv2.line(houghResult,(x1,y1),(x2,y2),(0,255,0),2)
    except Exception as e:
        print e

    # show result
    cv2.imshow('lineBottom',lineBottom)
    cv2.imshow('houghResult ',houghResult)


# exit
cv2.destroyAllWindows()

Is this a feasible approach ? If so, what's the correct way of doing line fitting in OpenCV Python ?

Otherwise, that's the best way to tackle this problem ?

Update Following Miki's advise I've tried OpenCV 3's LSD and got nicer results than with HoughLinesP but it looks like there's still some tweaking needed, although it doesn't look other than cv2.createLineSegmentDetector there aren't many options to play with:

I did something very similar with approxPolyDP, eventually merging almost collinear subsequent segments / short segments. — Miki, Aug 10 '17 at 16:41
Another approach is to use LSD (line segment detector)... I never used it but should worth a try — Miki, Aug 10 '17 at 16:54
@Miki that sounds interesting. I started with approxPolyDP, but didn't know what to do with it next. So you basically looped through lines and if the angle difference between pairs of lines was bellow a threshold you merged them into one line, right ? — George Profenza, Aug 10 '17 at 16:55
@Miki Is [this](http://docs.opencv.org/trunk/db/d73/classcv_1_1LineSegmentDetector.html#a1816a3c27f7c9b8d8acffec14451d4c4) what the LSD you mean ? (Gotta get OpenCV 3.0 Python setup in this case, still on 2.4.13 so far) — George Profenza, Aug 10 '17 at 16:57
Yes. First find a suitable epsilon for the approximation. You should end up with long segments for the straight parts, and shorter segments for the curved parts. You can then merge two subsequent segments if they're almost collinear to get a single segment for the straight parts, and remove short / not collinear segments in the curved parts. — Miki, Aug 10 '17 at 16:59

score 1 · Answer 1 · answered Aug 10 '17 at 17:57

1

It may be convenient to use curvature to find line segments. Here example of splitting contour by minimal curvature points, it may be better to use maximal curvature points in your case. B You can split your curve to parts, then each part approximate with line segment using RANSAC method.

answered Aug 10 '17 at 17:57

Andrey Smorodov

10,649
2
35
42

That looks very interesting, thanks for that (+1)! I'll port my little demo c++ and see how it goes. – George Profenza Aug 10 '17 at 19:55

score 1 · Answer 2 · answered Jul 10 '20 at 15:15

I know this is old but I found this after having a similar problem The method I used (after finding binary image) was along the lines of:

Find ends (points with fewest neighbors)
skeletonize (optional)
Starting at one end find a few closest points using skimage cdist
Perform a linear regression with these points and find all points in the image within a few pixels error of the line of best fit. I used query_ball_point
This gives additional points within the same straight line. Order them by distance from the last fiducial point. Some of these might be projections of the line onto distant parts of the object and should be deleted.
Repeat steps 4 and 5 until no more points are added.
Once no more points are added to the line you find the start of the next valid line by looking at R-squared for the fit. The line should have very high R squared eg. > 0.95 (depending on the image - I was getting > 0.99). Keep changing starting point until high R squared is achieved.
This gives a bunch of line segments from where it should be easy to find the angles between them. One potential problem occurs when the segment is vertical (or horizontal) and the slope becomes infinite. When this occurred I just flipped the axes around. You can also get around this by defining end points of a line and finding all points within a threshold distance from that line rather than doing the regression.

This involves a lot more coding than using the other methods suggested but execution time is fast and it gives much greater control over what is happening.

score 0 · Answer 3 · answered Aug 17 '17 at 08:36

Once you have the contour, you can analyze it using a method like the one proposed in this paper: https://link.springer.com/article/10.1007/s10032-011-0175-3

Basically, the contour is tracked calculating the curvature at each point. Then you can use a curvature threshold to segment the contour into straight and curved sections.

How to segment bent rod for angle calculations?

3 Answers3