Robust algorithm to find direction of arrow OpenCV

Question

The arrows I'm trying to process looks like this:

Filtered

Unfiltered

How can I determine the angle the arrow is pointing?

What I've tried: FindContour + ApproxPoly - This returns a triangle essentially. However the problem is figuring which corner is the "front".

Somehow, we should be able to detect that one of the edges is unique (it has a bump). Maybe test for axis of symmetry?

Another idea is maybe using image moments?

In my experience you can spend a lot of time fiddling with opencv's filters, edge detectors, line finders, etc... and make something that works. But at this point in time, just go train a CNN to do it ... it will work better and be less of a hack. Create a CNN classifier on the direction of the arrow. — wcochran, May 29 '21 at 17:32
Idea: Maybe, calculating the covariance of the pixels inside the triangle should yield the direction. Some transformation should be required to get it in terms of angle. — saveearth, May 29 '21 at 17:34
@saveearth Could you link some sort of article or explanation? — , May 29 '21 at 17:47
Checkout my solution here https://stackoverflow.com/a/66957554/10333818. There are other solutions in that question that you might find more helpful. In your specific case, try using centroid, which, from the images it seems is going to be tilted towards the base of the arrow (due to the bump). Triangle point that is longest distance from this then is the direction of the arrow. — Knight Forked, May 29 '21 at 18:32
Another idea is -> findContours (using cv2.RETR_CCOMP and using hierarchy-2 contours) -> convexHull -> 4 edges -> arrow pointing direction is the line between the common points between the 2 larger line segments and the 2 smaller line segments. — Knight Forked, May 29 '21 at 18:39

score 0 · Answer 1 · answered May 29 '21 at 18:51

You will need to do some steps. First, get the coordinates of the white pixels of the arrow, probably to have red, green and blue values over 200, you may need to inspect this for your images and make sure there is no extra pixels. Second, get the center of area of these pixels. Third, get the pixels of the black area inside the arrow that is around the previous center and get its own center. Finally get the properties of area of the black arrow Ix, Iy, and Ixy, which can be used to get the angle of rotation of the arrow, assuming up direction is zero.
You will need to do simple modifications to this angle, if it is a negative value, add Pi to it, also you will need to get the furthest point of arrow from its center, and if it right from the centroid, the angle should be greater than Pi, otherwise it should be less than it.

int i, j, x, y, t;
double xc, yc, Ix, Iy, Ixy, xf, yf, d, ang;
Bitmap img = new Bitmap(path);   // load only the area that contains the arrow
int[] dx, dy;
bool[,] b;
List<int> px, py;
px = new List<int>();
py = new List<int>();
dx = new int[] { -1, 1, 0, 0 };
dy = new int[] { 0, -1, 0, 1 };
b = new bool[img.Width, img.Height];
xc = yc = Ix = Iy = Ixy = xf = yf = d = t = 0;
for (x = 0; x < img.Width; x++)
    for (y = 0; y < img.Height; y++)
        if (img.GetPixel(x, y).R > 200 && img.GetPixel(x, y).G > 200 && img.GetPixel(x, y).B > 200)  // you will have to check this condition for your images
        {
            t++;        // get the number of pixels of the white arrow
            xc += x;
            yc += y;    
        }
// get the center of area of the white arrow
xc /= t;
yc /= t;
// start with that center to get the black arrow
px.Add((int)Math.Round(xc, 0));
py.Add((int)Math.Round(yc, 0));
b[px[0], py[0]] = true;
xc = px[0];
yc = py[0];
t = 1;
for (i = 0; i < px.Count; i++)
    for (j = 0; j < 4; j++)
    {
        x = px[i] + dx[j];
        y = py[i] + dy[j];
        if (x >= 0 && x < img.Width && y >= 0 && y < img.Height && !b[x, y] && img.GetPixel(x, y).R < 20 && img.GetPixel(x, y).G < 20 && img.GetPixel(x, y).B < 20)
        {
            t++;        // get the number of pixels of the black arrow
            xc += x;
            yc += y;
            px.Add(x);  // store x-coordinates of all arrow pixels
            py.Add(y);  // store y-coordinates of all arrow pixels
            b[x, y] = true;
        }
    }
// get the center of area of the black arrow
xc /= t;
yc /= t;
// calculate the properties of area
for (i = 0; i < t; i++)
{
    if (Math.Pow(px[i] - xc, 2) + Math.Pow(py[i] - yc, 2) > d)
    {
        xf = px[i] - xc;
        yf = py[i] - yc;
        d = Math.Pow(xf, 2) + Math.Pow(yf, 2);
    }
    Ix += Math.Pow(py[i] - yc, 2);
    Iy += Math.Pow(px[i] - xc, 2);
    Ixy += (px[i] - xc) * (py[i] - yc);
}
//  calculate the angel
ang = Math.Atan2(-2 * Ixy, Ix - Iy) / 2;
//  correct the angle
if (ang < 0)
    ang += Math.PI;
if (xf > 0 && ang < Math.PI)
    ang += Math.PI;
if (xf < 0 && ang > Math.PI)
    ang -= Math.PI;

I tried this on your images and got 0.311 rad for the first one and 0.699 rad for the second one.

Robust algorithm to find direction of arrow OpenCV

1 Answers1