@Peter de Rivaz offers a fundamental insight, albeit with no proof: the point is inside a loop iff it is inside the hull of formed by the outer vertices of the graph. We can prove this by proving:
- Any point inside the hull is inside a loop
- Any point outside the hull is not inside any loop
The first is easy to prove: any point inside the hull is inside a loop, because the hull itself is a loop.
The second can be proved by reductio ad absurdum. Very informally, for a point outside outside the hull to be inside a loop, there needs to be at least a vertex outside the hull, and for it to form a loop with at least two other vertices inside the loop, such that the point is inside that same loop. However, there can't be any vertices outside the loop, because, by definition, all vertices are inside it. Thus, by reductio ad absurdum, there are no points outside the hull that are inside any loop.
Now that we are sure we have a correct way to test what we want, we still need an algorithm that can tell whether a point is inside the hull. This may be accomplished through a simple extension of the ray casting algorithm.
Basically, we start with the list of all vertices, sorted by the vertical coordinate. Then, for each pair of successive vertices, we "create" a horizontal line that goes between them, and check what is the first and last edge that the line intersects. Those two edges are part of the hull. If the tested point is between any of these two edges, it will be inside the hull.
Here's a graphical representation of the first 3 iterations:
