I am writing a simple graphics editor where a user can draw a triangle (in either clockwise or counterclockwise orientation) and can then select the triangle and drag it around.
My issue comes down to finding whether or not the mouse cursor coordinates intersect with the triangle. Because I am using GLFW, my window space is defined on a Cartesian coordinate system where x/y have range [-1, 1].
This causes an issue when I try to determine whether I have an intersection using barycentric coordinates (or any other method found here)
My current approach is as follows:
double xpos, ypos;
glfwGetCursorPos(window, &xpos, &ypos);
// Get the size of the window
int width, height;
glfwGetWindowSize(window, &width, &height);
// Convert screen position to world coordinates
xworld = ((xpos / double(width)) * 2) - 1;
yworld = (((height - 1 - ypos) / double(height)) * 2) - 1; // NOTE: y axis is flipped in glfw
double area = 0.5 * (-p1y * p2x + p0y * (-p1x + p2x) + p0x * (p1y - p2y) + p1x * p2y);
double s = 1 / (2 * area) * (p0y * p2x - p0x * p2y + (p2y - p0y) * xworld + (p0x - p2x) * yworld),
t = 1 / (2 * area) * (p0x * p1y - p0y * p1x + (p0y - p1y) * xworld + (p1x - p0x) * yworld);
if (s > 0 && t > 0 && 1 - s - t > 0)
return true;
return false;
Which kinda works if triangle in the first quadrant and oriented counter-clockwise, however, it also recognizes as intersection if to the left of the triangle.
Thanks for any help.
EDIT: Issue was a typo I had in my code (wrong value for a vertice in my triangle) Ended up using the calculation of area approach to detect intersection.