4

I am a relative newcomer to image processing and this is the problem I'm facing - Say I have the image of an application form, like this:enter image description here

Now I would like to detect the locations of all the locations where data is to be entered. In this case, it would be the rectangles divided into a number of boxes like so(not all fields marked):

enter image description here

I can live with the photograph box also being detected. I've tried running the squares.cpp sample in the OpenCV sources, which does not quite get me what I want. I also tried the modified version here - the results were worse(my use case is definitely very different from the OP's in that question).

Also, Hough transforming to get the lines is not really working with/without blur-threshold as the noise in scanned image is contributing to extraneous lines, and also, thresholding is taking away parts of the combs(the small squares), and hence the line detection is not up to the mark.

Note that this form is not a scanned copy of a printed form, but the real input might very well be a noisy, scanned image of a printed form.

While I'm definitely sure that this is possible(at least with some tolerance allowed) and I'm trying to get at the solution, it would be really helpful if I get insights and ideas from other people who might have tried something like this/enjoy hacking on CV problems. Also, it would be really nice if the answers explain why a particular operation was done (e.g., dilation to try and fill up any holes left by thresholding, etc)

Community
  • 1
  • 1
yati sagade
  • 1,355
  • 9
  • 24
  • do you know the format of the form ahead of time? – Hammer Nov 10 '12 at 18:19
  • no - All I know is that data will be entered in such boxes and that the colour of the interiors of the boxes will be brighter than or same as the background colour. – yati sagade Nov 10 '12 at 18:27

1 Answers1

1

Are the forms consistent in any way? Are the "such boxes" the same size on all forms? If you can rely on a consistent size, like the character boxes in the form above, you could use template matching.

Otherwise, the problem seems to be: find any/all rectangles on the image (with a post processing step to filter out any that have a significant amount of markings within, or to merge neighboring rectangles).

The more you can take advantage of the consistencies between the forms, the easier the problem will be. Use any context you can get.

EDIT

Using the gradients (computed by using a Sobel kernel in both the x and the y direction) you can weed out a lot of the noise.

Using both you can find the direction of the gradients (equation can be found here: en.wikipedia.org/wiki/Sobel_operator). Let's say we define a discriminating feature of a box to be a vertical or horizontal gradient. If the pixel's gradient has an orientation that's either straight horizontal or straight vertical, keep it, set all else to white.

To make this more robust to noise, you can use a sliding window (3x3) in which you compute the median orientation. If the median (or mean) orientation of the window is vertical or horizontal, keep the current (middle of the window) pixel, otherwise set it to white.

You can use OpenCV for the gradient computation, and possibly the orientation/phase calculation, but you'll probably need to write the code it do the actual sliding window code. I'm not intimately familiar with OpenCV

Noremac
  • 3,445
  • 5
  • 34
  • 62
  • No, I cannot rely on "consistencies" across various forms, as the forms might come from varying sources not known ahead of time. As I mentioned in the question, finding ALL the rects (the squares.cpp sample really searches also for rectangles, not just squares) in the image looks like one way to go, but the results aren't very good :| – yati sagade Nov 14 '12 at 07:18
  • I'm curious what you mean by "aren't very good". Does it get all of the squares with some extras? Does it not get all of them? Either way because of the generality of the problem, I highly doubt you'll be able to simply use an existing program and will need to post process to either weed-out or join boxes. – Noremac Nov 14 '12 at 15:07
  • Noremac, It doesn't get all or even most of the squares in the image. I was playing with Sobel filters in the horizontal and vertical directions to get lines in those directions, which does a really great job. But now the task is to extract the squares, now that the unnecessary edges given by Canny are not there. Theere is a lot of noise, anyway and the edges from the Sobel output are very faint - but geometrically good nevertheless. – yati sagade Nov 15 '12 at 08:52
  • That could be a good direction to go in, using the gradients (from the Sobel kernel) in both the x and the y direction you can weed out a lot of the noise. Using both you can find the direction of the edges (equation can be found here: http://en.wikipedia.org/wiki/Sobel_operator) and weed out gradients that do not match the range of angles you want (i.e. 0, pi/2, pi, 3pi/2 with some looseness to account for noise). Additionally you'd probably want to do a sliding window in which you determine to keep a pixel/edge based on its neighbors, ensuring that they are consistent. – Noremac Nov 15 '12 at 14:09
  • Noremac, I was wondering if you could elaborate with an answer or an edit to this answer. That would be of immense help :) Using OpenCV libraries would be a plus, but not a requirement, I can code the thing up if needed. – yati sagade Nov 15 '12 at 16:32
  • What? you don't like it all jumbled together with no line breaks? :). I'm updated my answer above and gave some more detail. I've used openCV, but not that much, so I don't know if OpenCV has a straight forward approach to this. – Noremac Nov 15 '12 at 19:36