How can I detect TV Screen from an Image with OpenCV or Another Library?

Question

I've working on this some time now, and can't find a decent solution for this.

I use OpenCV for image processing and my workflow is something like this:

1. Took a picture of a tv.
2. Split image in to R, G, B planes - I'm starting to test using H, S, V too and seems a bit promising.
3. For each plane, threshold image for a range values in 0 to 255
4. Reduce noise, detect edges with canny, find the contours and approximate it.
5. Select contours that contains the center of the image (I can assume that the center of the image is inside the tv screen)
6. Use convexHull and HougLines to filter and refine invalid contours.
7. Select contours with certain area (area between 10%-90% of the image).
8. Keep only contours that have only 4 points.

But this is too slow (loop on each channel (RGB), then loop for the threshold, etc...) and is not good enought as it not detects many tv's.

My base code is the squares.cpp example of the OpenCV framework.

The main problems of TV Screen detection, are:

1. Images that are half dark and half bright or have many dark/bright items on screen.
2. Elements on the screen that have the same color of the tv frame.
3. Blurry tv edges (in some cases).

I also have searched many SO questions/answers on Rectangle detection, but all are about detecting a white page on a dark background or a fixed color object on a contrast background.

My final goal is to implement this on Android/iOS for near-real time tv screen detection. My code takes up to 4 seconds on a Galaxy Nexus.

Hope anyone could help . Thanks in advance!

Update 1: Just using canny and houghlines, does not work, because there can be many many lines, and selecting the correct ones can be very difficult. I think that some sort of "cleaning" on the image should be done first.

Update 2: This question is one of the most close to the problem, but for the tv screen, it didn't work.

Answer 1

Hopefully these points provide some insight:

1)

If you can properly segment the image via foreground and background, then you can easily set a bounding box around the foreground. Graph cuts are very powerful methods of segmenting images. It appears that OpenCV provides easy to use implementations for it. So, for example, you provide some brush strokes which cover "foreground" and "background" pixels, and your image is converted into a digraph which is sliced optimally to split the two. Here is a fun example:

http://docs.opencv.org/trunk/doc/py_tutorials/py_imgproc/py_grabcut/py_grabcut.html

This is a quick something I put together to illustrate its effectiveness:

2)

If you decide to continue down the edge detection route, then consider using Mathematical Morphology to "clean up" the lines you detect before trying to fit a bounding box or contour around the object.

http://en.wikipedia.org/wiki/Mathematical_morphology

3)

You could train across a dataset containing TVs and use the viola jones algorithm for object detection. Traditionally it is used for face detection but you can adapt it for TVs given enough data. For example you could script downloading images of living rooms with TVs as your positive class and living rooms without TVs as your negative class.

http://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework http://docs.opencv.org/trunk/doc/py_tutorials/py_objdetect/py_face_detection/py_face_detection.html

4)

You could perform image registration using cross correlation, like this nice MATLAB example demonstrates:

http://www.mathworks.com/help/images/examples/registering-an-image-using-normalized-cross-correlation.html

As for your template TV image which would be slid across the search image, you could obtain a bunch of pictures of TVs and create "Eigenscreens" similar to how Eigenfaces are used for facial recognition and generate an average TV image:

http://jeremykun.com/2011/07/27/eigenfaces/

5)

It appears OpenCV has plenty of fun tools for describing shape and structure features, which appears to be mainly what you're interested in. Worth a look if you haven't seen this already:

http://docs.opencv.org/modules/imgproc/doc/structural_analysis_and_shape_descriptors.html

Best of luck.