4

I'm trying to detect shapes written on a whiteboard with a black/blue/red/green marker. The shapes can be circles, rectangles or triangles. The image can be found at the bottom of this post.

I'm using OpenCV as the framework for the image recognition.

My first task is to research and list the different strategies that could be used for the detection. So far I have found the following:

1) Grayscale, Blur, Canny Edge, Contour detection, and then some logic to determine if the contours detected are shapes?

2) Haar training with different features for shapes

3) SVM classification

4) Grayscale, Blur, Canny Edge, Hough transformation and some sort of color segmentation?

Are there any other strategies that I have missed? Any newer articles or tested approaches? How would you do it?

One of the test pictures: https://drive.google.com/file/d/0B6Fm7aj1SzBlZWJFZm04czlmWWc/view?usp=sharing

UPDATE: The first strategy seems to work the best, but is far from perfect. Issues arise when boxes are not closed, or when the whiteboard has a lot of noise. Haar training does not seems very effective because of the simple shapes to detect without many specific features. I have not tried CNN yet, but it seems most appropriate to image classification, and not so much to detect shapes in a larger image (but I'm not sure)

MortenGR
  • 803
  • 1
  • 8
  • 22
  • Have a look at [Detecting lines and shapes in OpenCV](http://stackoverflow.com/a/32077775/5008845) and also [How to make the blackboard text appear clearer](http://stackoverflow.com/a/19971599/5008845) – Miki Sep 22 '16 at 08:20

1 Answers1

1

I think that the first option should work. You can use fourier descriptors in order to classify the segmented shapes.

http://www.isy.liu.se/cvl/edu/TSBB08/lectures/DBgrkX1.pdf

Also, maybe you can find something useful here:

http://www.pyimagesearch.com/2016/02/08/opencv-shape-detection/

If you want to try a more challenging but modern approach, consider deep learning approach (I would start with CNN). There are many implementations available on the internet. Although it is probably an overkill for this specific project, it might help you in the future...

GpG
  • 502
  • 5
  • 11
  • I think my biggest problem right now is that a whiteboard is not always white. There can be reflections, old drawings (that was not erased completely) and general noise that exist even after I apply some blurring effect. The best result right now is from blur, canny edge, close morphing, contours and then some logic to discard invalid contours. I will look into CNN. Do you have some tutorial / hello world example that I can use to get started? – MortenGR Sep 26 '16 at 06:26
  • here you will find a ton of useful information: https://github.com/ChristosChristofidis/awesome-deep-learning. In python, you have two great libraries (keras and lasagne) that implement different approaches of deep learning. – GpG Sep 26 '16 at 13:08
  • if you can, upload the example image. – GpG Sep 26 '16 at 13:09
  • I have added the image link to the post (https://drive.google.com/file/d/0B6Fm7aj1SzBlZWJFZm04czlmWWc/view?usp=sharing). I want to detech the two rectangles and the circle. It's ok if the top box is detected as well. – MortenGR Sep 26 '16 at 13:24
  • the marker color can be only red, green, blue or black. If you take this into the account you will be able to get rid of reflections and shadings. Why not just separately threshold each color chanel of the image additionaly with the grey image and combine the results. That way, you should be left with only red, green, blue and black text with some minor additional effects. Next dilation, labeling and classification. – GpG Sep 27 '16 at 09:20
  • I will try it. I tried some histogram filtering using all three colors at once but it was not very successful :). So: 1) Split 2) Threshold according to color 3) Merge (not quite sure how to merge the channels) 4) Blur etc 5) Canny 6) Close Morphing 7) Contour detection – MortenGR Sep 27 '16 at 09:22
  • I think there is no need for canny. With thresholding you will already get binary image. Maybe instead of dilation, closing would be more appropriate. – GpG Sep 27 '16 at 09:27
  • using convex hull might be useful. Check out this link: http://stackoverflow.com/questions/31974843/detecting-lines-and-shapes-in-opencv-using-python?noredirect=1&lq=1 – GpG Sep 27 '16 at 12:48
  • I had very little success with filtering the colors. My filters kept leaving bright white marks that resulted in false edges/contours. Using convex hull on the other hand, seems very promising The challenge now is that sometimes a square and an arrow is considered a single shape. I will create a seperate question for this :) – MortenGR Sep 28 '16 at 08:14