I think you would be better off trying to compare the distance from the actual point to a comparable point on the line. The closest comparable point would be the one that forms a perpendicular line like this. a is your point in question and b is the comparable point on the line line between the two vertices that you will check distance to.
However there's another method which I think might be more optimal for this case (as it seems most of your test points lie pretty close to the desired line already). Instead of find the perpendicular line point we can simply check the point on the line that has the same X value like this. b in this case is a lot easier to calculate:
X = a.X - 0.X;
Slope = (1.Y - 0.Y) / (1.X - 0.X);
b.X = 0.X + X;
b.Y = 0.Y + (X * Slope);
And the distance is simply the difference in Y values between a and b:
distance = abs(a.Y - b.Y);
One thing to keep in mind is that this method will become more inaccurate as the slope increases as well as become infinite when the slope is undefined. I would suggest flipping it when the slope > 1 and checking for a b that lies at the same y rather than x. That would look like this:
Y = a.Y - 0.Y;
Inverse_Slope = (1.X - 0.X) / (1.Y - 0.Y);
b.Y = 0.Y + Y;
b.X = 0.Y + (Y * Inverse_Slope);
distance = abs(a.X - b.X);
Note: You should also check whether b.X is between 0.X and 1.X and b.Y is between 0.Y and 1.Y in the second case. That way we are not checking against points that dont lie on the line segment.
I admit I don't know the perfect terminology when it comes to this kind of thing so it might be a little confusing, but hope this helps!