OpenCV Canny + Watershed

Question

I'm using a canny edge detection and a finding contours function (both OpenCV) to create markers for the watershed transform. Everything works fine but I'm not 100% satisfied with the results. The reason is that some edges are missing and therefore important information is lost. In more detail, I got a bunch of windows (front views), which are rectangles, after the watershed transform I end up with something like this:

but I would rather have nice rectangles, that are complete and not open to one side. While maintaining irregular shapes (bushes in front of the house, cars..) Any ideas how I could solve this problem?I thought about overlaying the whole image with a grid, but I can't make it work.

Thank you very much.

Here is my code:

Mat gray;
cvtColor(im, gray, CV_BGR2GRAY);

// Use Canny instead of threshold to catch squares with gradient shading
Mat bw;
Canny(gray, bw, 0, 100, 5, true);

// Find contours
vector<vector<Point> > contours;
vector<Vec4i> hierarchy;

findContours( bw, contours, hierarchy,
    CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE );


// watershed
Mat markers(bw.size(), CV_32S);
markers = Scalar::all(0);
int idx = 0;
int compCount = 0;
for( ; idx >= 0; idx = hierarchy[idx][0], compCount++ ) {
    if (fabs(contourArea(contours[compCount])) < min_size )
        continue;
    drawContours(markers, contours, idx, Scalar::all(compCount+1), 1, 8, hierarchy, INT_MAX);
}
watershed( im, markers );

As requested, here is the original image, the image I would like to get and my output:

And I would like to have a segmentation like this (although over segmentation does not hurt, I just need to make sure, I get all the details):

While I get something like this: (please ignore the colours, they are not important for this question and are just a result of my overall program). This is only one example, if you want, I can show you more, also please have a look at the etrims dataset, all my pictures are from there.

Answer 1

Two things -

1) As already mentioned, edge detection results in spurious edges being picked up.

2) Using these edges as markers for watershed segmentation results in over-segmentation because every marker produces a segmented region in the output.

Strategy -

(i) Preprocessing: Smooth the image heavily (morphological opening by reconstruction can be used for homogenizing the intensities without significantly affecting edges you are interested in).

(ii) Markers: Instead of using edges as seeds, I'd use the local extrema. Ideally, we want one marker for every region we want segmented.

(iii) Segmentation: Find the gradient magnitude (range filtering is also a good option) of the image from step (i) and use that as the segmentation function.

Using this strategy, I get the following segmentation.

Alternatively, after step (i), you can use Canny edge detection and do some morphological cleanup (to fill contours and remove edges that remain). This is what I get.

These are not exactly the expected segmentation (some objects like the car are not detected), but are a good start.

Edit: The MATLAB code used to generate the images -

% convert to grayscale
img = rgb2gray(origImg);

% create an appropriate structuring element
w_size = 20;
seSquare = strel('square', w_size);

% opening by reconstruction - to smooth dark regions
imgEroded = imerode(img, seSquare);
imgRecon = imreconstruct(imgEroded, img);

% invert and repeat - to smooth bright regions
imgReconComp = imcomplement(imgRecon);
imgEroded2 = imerode(imgReconComp, seSquare);
imgRecon2 = imreconstruct(imgEroded2, imgReconComp);

% get foreground markers
fgm = imregionalmax(imgRecon2);

% get background markers - this step can be skipped 
% in which case only fgm would be the marker image 
% and the segmentation would be different 
distTrans = bwdist(fgm);
wLines= watershed(distTrans);
bgm = wLines == 0;

% get the segmentation function and impose markers
% perform watershed segmentation
seSquare3 = strel('square', 3);
rangeImg = rangefilt(imgRecon2, getnhood(seSquare3));
segFunc = imimposemin(rangeImg, fgm | bgm);
grayLabel = watershed(segFunc);

rgbLabel= label2rgb(grayLabel);
figure, imshow(rgbLabel); title('Output using Watershed')

% alternatively, extract edges from the preprocessed image
% perform morph cleanup
bwEdges = edge(imgRecon2, 'canny');
bwFilled = imfill(bwEdges, 'holes');
bwRegions = imopen(bwFilled, seSquare3);
grayLabel = bwlabel(bwRegions);

rgbLabel = label2rgb(grayLabel, 'jet', 'k');
figure, imshow(rgbLabel); title('Output using Canny')

Answer 2

from the looks of the desired output and the program's output, it seems that the edge detector is finding spurious edges. Canny edge detector contains a low-pass filter, but it might help for you to do a separate Gaussian low-pass filtering step before you actually run the Canny edge detector.

Other than that, it is difficult to achieve the desired result. For e.g., look at the top-most windows in the picture. They have distinct colors --- the frame, the shadow of the frame, and the window. The boundaries of these colors will be detected as edges by the Edge detector.