I suggest following solution, which also works in case of a polygon that is around the north pole, where a calculation of the longitude and latitude difference doesn't make sense.
The solution transforms the longitude and latitude of points on the earth into three-dimensional coordinates by using the World Geodetic System 84. With those three-dimensional points, you can calculate the projection of one point on the line defined by two other points in three-dimensional space.
Here is the code doing the calculations. It uses the class javafx.geometry.Point3D
, available in Java 8.
/** Semi-major axis of earth in meter */
public static final double WGS84_A = 6378137.0;
/** Semi-minor axis of earth in meter */
public static final double WGS84_B = 6356752.314245;
/** Eccentricity of earth */
public static final double WGS84_E =
Math.sqrt( (WGS84_A * WGS84_A) / (WGS84_B * WGS84_B) - 1);
public static final double DEGREES_TO_RADIANS = Math.PI / 180;
/**
* Calculates a three-dimensional point in the
* World Geodetic System (WGS84) from latitude and longitude.
*/
public static Point3D latLonToPoint3D(double lat, double lon) {
double clat = Math.cos(lat * DEGREES_TO_RADIANS);
double slat = Math.sin(lat * DEGREES_TO_RADIANS);
double clon = Math.cos(lon * DEGREES_TO_RADIANS);
double slon = Math.sin(lon * DEGREES_TO_RADIANS);
double N = WGS84_A / Math.sqrt(1.0 - WGS84_E * WGS84_E * slat * slat);
double x = N * clat * clon;
double y = N * clat * slon;
double z = N * (1.0 - WGS84_E * WGS84_E) * slat;
return new Point3D(x, y, z);
}
/**
* Calculates distance of projection p of vector a on vector b.
*
* Use formula for projection, with p being the projection point:
* <p>
* p = a X b / |b|^2 * b
* </p>
* X being the dot product, * being multiplication of vector and constant
*/
public static Point3D calculateProjection(Point3D a, Point3D b) {
return b.multiply(a.dotProduct(b) / (b.dotProduct(b)));
}
/**
* Calculates shortest distance of vector x and the line defined by
* the vectors a and b.
*/
public static double calculateDistanceToLine(Point3D x, Point3D a, Point3D b) {
Point3D projectionOntoLine =
calculateProjection(x.subtract(a), b.subtract(a)).add(a);
return projectionOntoLine.distance(x);
}
By calling calculateDistanceToLine
with the point and the polygon segments' points, you are able to find the nearest line defined by the edge points and extended to infinity. In the case of a concave polygon, this may not be what you want, as you see in the picture.

Taking into account that the distance to the polygon edge must be at least as long as the distance to the nearest edge point, you can get the distance to the edge as:
Math.max(calculateDistanceToLine(x, edgePoint1, edgePoint2),
Math.min(x.distance(edgePoint1), x.distance(edgePoint2)));
Note that this calculation yields also not the distance on the surface of the earth, but the direct distance cutting through the earth. Anyway, it should suffice for choosing the shortest distance.
The function latLonToPoint3D
is a modified version of the function that I found here.