According the info and sample images provided, here is how I would approach this problem:
Prerequisite
As the red marks are lines created by the HoughLinesP
, you have the following information on each line: (x1, y1, x2, y2) where (x1, y1) is the start point and (x2, y2) the end point of the line.
For testing purpose, I created 3 lines:

Pipeline
- Create a list of points (
contour
in my code) containing the start and end points of each line.
- Compute the convex hull of the list of points using
convexHull
.
At this point you will get something in this taste:

Note that the generated hull (in green) does contain more than 4 points (that is, not only the extreme points). So what we'll do next is to simplify the hull in order to retain only the 4 extreme points.
- Simplify the hull with
approxPolyDP
- The 4 remaining points of the hull should be an approximation of the extreme points.
Here you can see the final result with the approximate hull in green and the 4 extreme points in yellow:

Python code
Finally, as a reference, here is the Python code I used (This shouldn't be to hard to port into Java):
import numpy as np
import cv2
input_img = np.zeros((400,400,3), np.uint8)
# 3 lines added for testing purpose
lines = []
lines.append(((350,350),(250,200))) # (start point, end point)
lines.append(((50,350),(55,330)))
lines.append(((60,250),(80,200)))
contour = [] #in Java you'd use a List of Points
for line in lines:
# we create a contour with the start and end points of each line
contour.append(line[0])
contour.append(line[1])
# we draw the lines (in your case, detected by HoughLinesP)
cv2.line(input_img,line[0],line[1],(0,0,255),3)
contour = np.array(contour)
hull = cv2.convexHull(contour) # we compute the convex hull of our contour
epsilon = 0.05*cv2.arcLength(hull,True) # epsilon may need to be tweaked
approx_hull = cv2.approxPolyDP(hull,epsilon,True)
# we check that there are only 4 points left in the approximate hull (if there are more, then you'd need to increase epsilon
if(len(approx_hull) == 4):
for final_point in approx_hull :
# we draw the 4 points contained in the approximate hull
cv2.circle(input_img, (final_point[0][0], final_point[0][1]),5,(0,242,255),2)
cv2.drawContours(input_img, [approx_hull], 0, (0,255,0),1)
cv2.imshow("result",input_img)
Alternative
I think this solution from @YvesDaoust could work aswell. I didn't have time to test it so please, keep me updated if you do. :)