The two existing answers have large shortcomings:
Durkee's method only works if the distances between subsequent points is exactly identical. Points must have coordinates perfectly representable as floating-point values, so that the distances between subsequent points on a line can be found to be identical. If points are not equidistant, the method does nothing. Also, the start and end of the polygon are not examined together, so if a straight line is formed across the start/end of the polygon, one point too many will remain.
ShadowMan's method is better in that distances do not need to be identical, and the line across the start/end of the polygon is correctly handled. However it also uses floating point equality comparisons, which will not work in general. Only with integer coordinates will this method work correctly. Furthermore it uses vecnorm
(which does a square root) and a division, both are relatively expensive operations (when compared to the method shown here).
To see if three points form a straight line, a simple arithmetic rule can be used. Let's say we have points p0
, p1
and p2
. The vector from p0
to p1
and the vector from p0
to p2
form the basis of a parallelogram, whose area can be computed by the cross product of the two vectors (in 2D, the cross product is understood to use z=0
, with the resulting vector having x=0
and y=0
, only the z
value is useful; thus, the 2D cross product we assume to produce a scalar value). It can be computed as follows:
v1 = p1 - p0;
v2 = p2 - p0;
x = v1(1)*v2(2) - v1(2)*v2(1);
x
, the cross product, will be zero if the two vectors are parallel, meaning that the three points are collinear. But the test for equality to 0 must have some tolerance, since floating-point arithmetic is inexact. I use 1e-6 as a tolerance here. Use a value that is several orders of magnitude smaller than the distance between your points.
Given an input set of points p
, we can find the corner points with:
p1 = p; % point 1
p0 = circshift(p1,1); % point 0
v1 = p1 - p0; % vector from point 0 to 1
v2 = circshift(p1,-1) - p0; % vector from point 0 to 2
x = v1(:,1).*v2(:,2) - v1(:,2).*v2(:,1); % cross product
idx = abs(x) > 1e-6; % comparison with tolerance
p = p(idx,:); % corner points
Do note that this cross product test will fail if two consecutive points have identical coordinates (i.e. one of the vectors has a zero length). An additional test would be necessary if the data could have duplicated points.
Here are the results of the three methods. I've created a polygon with non-trivial coordinates and not equally spaced vertices. I've also put the start/end gap in the middle of a straight edge. These characteristics are purposeful to show the shortcomings of the other two methods.

This is the code I used to produce the graph:
% Make a polygon that will be difficult for the other two methods
p = [0,0 ; 0.5,0 ; 1,0 ; 1,1 ; 0.5,1 ; 0,1];
p = p + rand(size(p))/3;
p(end+1,:) = p(1,:);
q = [];
for ii = 1:size(p,1)-1
t = p(ii,:) + (p(ii+1,:) - p(ii,:)) .* [0;0.1;1/3;0.45;0.5897545;pi/4;exp(1)/3];
q = [q;t];
end
q = circshift(q,3,1);
figure
subplot(2,2,1)
plot(q(:,1),q(:,2),'bo-')
axis equal
title('input')
subplot(2,2,2)
res1 = method1(q);
plot(res1(:,1),res1(:,2),'ro-')
axis equal
title('Durkee''s method')
subplot(2,2,3)
res2 = method2(q);
plot(res2(:,1),res2(:,2),'ro-')
axis equal
title('ShadowMan''s method')
subplot(2,2,4)
res3 = method3(q);
plot(res3(:,1),res3(:,2),'go-')
axis equal
title('correct method')
% Durkee's method: https://stackoverflow.com/a/55603145/7328782
function P = method1(P)
a = logical([1 diff(P(:,1),2)' 1]);
b = logical([1 diff(P(:,2),2)' 1]);
idx = or(a,b);
P = P(idx,:);
end
% ShadowMan's method: https://stackoverflow.com/a/55603040/7328782
function corners = method2(poly0)
poly0Z = circshift(poly0,1);
poly0I = circshift(poly0,-1);
unitVectIn =(poly0 - poly0I)./vecnorm((poly0 - poly0I),2,2);
unitVectOut =(poly0Z - poly0)./vecnorm((poly0Z - poly0),2,2);
cornerIndices = sum(unitVectIn == unitVectOut,2)==0;
corners = poly0(cornerIndices,:);
end
% vecnorm is new to R2017b, I'm still running R2017a.
function p = vecnorm(p,n,d)
% n is always 2
p = sqrt(sum(p.^2,d));
end
function p = method3(p1)
p0 = circshift(p1,1);
v1 = p1 - p0;
v2 = circshift(p1,-1) - p0;
x = v1(:,1).*v2(:,2) - v1(:,2).*v2(:,1);
idx = abs(x) > 1e-6;
p = p1(idx,:);
end