Guarantee outward direction of polygon normals

Question

I'm trying to write a 2D game in Java that uses the Separating Axis Theorem for collision detection. In order to resolve collisions between two polygons, I need to know the Minimum Translation Vector of the collision, and I need to know which direction it points relative to the polygons (so that I can give one polygon a penalty force along that direction and the other a penalty force in the opposite direction). For reference, I'm trying to implement the algorithm here.

I'd like to guarantee that if I call my collision detection function collide(Polygon polygon1, Polygon polygon2) and it detects a collision, the returned MTV will always point away from polygon1, toward polygon2. In order to do this, I need to guarantee that the separating axes that I generate, which are the normals of the polygon edges, always point away from the polygon that generated them. (That way, I know to negate any axis from polygon2 before using it as the MTV).

Unfortunately, it seems that whether or not the normal I generate for a polygon edge points towards the interior of the polygon or the exterior depends on whether the polygon's points are declared in clockwise or counterclockwise order. I'm using the algorithm described here to generate normals, and assuming that I pick (x, y) => (y, -x) for the "perpendicular" method, the resulting normals will only point away from the polygon if I iterate over the vertices in clockwise order.

Given that I can't force the client to declare the points of the polygon in clockwise order (I'm using java.awt.Polygon, which just exposes two arrays for x and y coordinates), is there a mathematical way to guarantee that the direction of the normal vectors I generate is toward the exterior of the polygon? I'm not very good at vector math, so there may be an obvious solution to this that I'm missing. Most Internet resources about the SAT just assume that you can always iterate over the vertices of a polygon in clockwise order.

score 2 · Answer 1 · edited May 23 '17 at 12:18

You can just calculate which direction each polygon is ordered, using, for example, the answer to this question, and then multiply your normal by -1 if the two polygons have different orders.

You could also check each polygon passed to your algorithm to see if it is ordered incorrectly, again using the algorithm above, and reverse the vertex order if necessary.

Note that when calculating the vertex order, some algorithms will work for all polygons and some just for convex polygons.

score 1 · Accepted Answer · edited May 23 '17 at 10:09

I finally figured it out, but the one answer posted was not the complete solution so I'm not going to accept it. I was able to determine the ordering of the polygon using the basic algorithm described in this SO answer (also described less clearly in David Norman's link), which is:

for each edge in polygon:
    sum += (x2 - x1) * (y2 + y1)

However, there's an important caveat which none of these answers mention. Normally, you can decide that the polygon's vertices are clockwise if this sum is positive, and counterclockwise if the sum is negative. But the comparison is inverted in Java's 2D graphics system, and in fact in many graphics systems, because the positive y axis points downward. So in a normal, mathematical coordinate system, you can say

if sum > 0 then polygon is clockwise

but in a graphics coordinate system with an inverted y-axis, it's actually

if sum < 0 then polygon is clockwise

My actual code, using Java's Polygon, looked something like this:

//First, find the normals as if the polygon was clockwise

int sum = 0;
for(int i = 0; i < polygon.npoints; i++) {
    int nextI = (i + 1 == polygon.npoints ? 0 : i + 1);
    sum += (polygon.xpoints[nextI] - polygon.xpoints[i]) * 
        (polygon.ypoints[nextI] + polygon.ypoints[i]);
}

if(sum > 0) {
    //reverse all the normals (multiply them by -1)
}

Guarantee outward direction of polygon normals

2 Answers2