You want to check if your shape is a quadrilateral, in particular a convex quadrilateral (look for concave and convex polygons). For the two problems that you showed, you can used two tests:
For the first case you just need to check that no interior angle between two sides is higher than 180 degrees, i.e., that your polygon is convex. To do this, check this question in SO.
I am not sure that checking for "convexitivity" solves the second case. You should try. If it doesn't, you need to check that the sum of the four interiror angles is 360 degrees (this holds true for all quadrilaterals). To get the angle between two segments you can convert them to vectors and use the dot product. The angle at vertex 2 would be:
v_{12} = ( x2 - x1 , y2 - y1 ) --> Vector from point 1 to point 2
v_{23} = ( x3 - x2 , y3 - y2 ) --> Vector from point 2 to point 3
Now calculate the dot product
v_{12} . v_{23} = (x3 - x2)(x2 - x1) + (y3 - y2)(y2 - y1)
The cosinus of the angle between segments 12 and 23 is the dot product divided by the modulus of each vector.