Imposing limitations on 4-point image transformation

Question

I'm working on a 4-point image transformation for an application. The user would either drag each corner around to create a valid quadrilateral shape, or use its "bounding box" to resize it vertically, horizontally, and proportionally.

So far, I have the image transformation part working. However, I'm having a hard time imposing limitations to it so that...

1. The vertices doesn't cross over each other. In another words, I don't allow the user to create an hour-glass shape. It must always be a quadrilateral.
2. Likewise, the angle between each corner must be greater than 0. Otherwise, the vertices will end up in a line.
3. It has a minimum size on this image. That is, each corner has to be a certain distance away from each other, and their opposing lines.
4. The user cannot "flip" the image to its backside. The 4 corners (p1, p2, p3, and p4) must appear in clockwise order.
5. Concave kite shapes and triangles are valid.

I was wondering if there was a formula or a paper regarding these issues. I do currently have the formulas for finding if 2 line segments intersect (and where), as well as finding the closest point on a line to another point. Most of my implementations hasn't been user-friendly as I liked, as the corners would jump all over the place when imposing restrictions.

P.S. I'm using C# for this project, with DirectX. The application is solely 2D, however.

Answer 1

My suggestion is not to impose restriction but just drawing say just a red wireframe polygon if for you it's unacceptable.

It's more or less like when programmers put restriction on form input fields e.g. begin date must come before end date and don't allow you to type in a date in the begin field that's after the date in the end field... forcing users to tab to end field first, then entering the end date, then tabbing back to the begin field to enter the correct begin date.

Just showing the fields in red is MUCH more usable... and actually requires less coding.

Trapping user mouse movements is rarely a sensible idea. If you can't apply what user is asking just make this evident instead of constraining the movement... may be the user is moving the first vertex and wants to move another vertex later to a position that will make your transform valid (exactly like the two date fields example).

If you really think you MUST prevent invalid positions at all times (and that for example will forbid the user from flipping the image even if your transform would allow a flipping operation without problems) then the simplest solution that comes to my mind is just having an isValid() function and just not moving the keypoint to the new position if it's invalid.

Having code that "slides" around invalid state space areas is IMO quite difficult to handle nicely, also because invalid areas are going to be quite close to "almost invalid" areas. Even in case "sliding" is a requirement I'd probably go for implementing an implicit isValid() approach by doing a local search around the keypoint position to find what is the closest valid point to the point the user is asking for.

Explicitly computing all sliding possibilities would be a nightmare to get correct and also to maintain if you need to change something in the transformation algorithm (and consequently in what is valid and what is not).

Answer 2

When you start out, and each time the user moves a handle, determine the set of horizontal and vertical lines that the other handles cannot pass, and enforce those boundaries during a drag. This will solve your first problem.

For the second, when the mouse moves during a drag, calculate the distance between the cursor and each of the other 3 handles; if it is less than your defined minimum distance then move the handle in a circular path around the other handle(s).