9

I'm using Sqlalchemy to define my tables and such and here is some code I came up with:

locations = Table('locations', Base.metadata,
Column("lat", Float(Precision=64), primary_key=True),
Column("lng", Float(Precision=64), primary_key=True),
)

I read somewhere that latitude and longitude require better precision than floats, usually double precision. So I set the precision manually to 64, is this sufficient? Overkill? Would this even help for my situation?

Community
  • 1
  • 1
john
  • 3,043
  • 5
  • 27
  • 48
  • 1
    "read somewhere". You should really find that reference and get an accurate quote from them. 1 degree of latitude is 60 nautical miles, about 364,567 feet. That means that 9 digits (3 to the left, 6 to the right) gets you to within one foot. That's about a single-precision floating-point number. PLease find the quote that said you needed more and provide a link or a reference. – S.Lott Jul 07 '11 at 00:54
  • The link appears to say 12 decimal places? That's 1.852E-9 m. That's about 2 nanometers. That seems ridiculous. – S.Lott Jul 07 '11 at 01:31
  • A single-precision float holds 6-7 decimal digits; if you want 9, you can't use single-precision, but double-precision is sufficient. – Jonathan Leffler Jul 07 '11 at 05:26
  • (Thanks @Jonathan Leffler) A 7.22 decimal digit single precision number would get you 4 digits to the right of the decimal point. 0.0001 degree is 11m, 36ft. The question remains: **What are your use cases?** How many digits do you need for your use cases? – S.Lott Jul 07 '11 at 10:57
  • I just want to store the lat/lng of business from Google maps and/or a user's location when they check in from their smartphone. – john Jul 07 '11 at 21:00
  • @S.Lott Keep in mind that no finite number of binary digits can [store a simple decimal number](https://stackoverflow.com/a/3448855/712526) like 0.1. Store as a float, use it as a coordinate, and you can easily be off by a kilometer. – jpaugh Sep 24 '17 at 13:10

4 Answers4

7

Nobody else here provided concrete numbers with proof for the worst-case accuracy of a floating point lat/long. I needed to know this for something I was working on, so here is my analysis in case it helps someone else.

A single-precision floating point offers 24-bits of precision in the significand (the binary exponential notation of a number). As the whole part of the number gets larger, the number of bits after the decimal goes down. Therefore, the worst-case accuracy for a latitude or longitude is when the magnitude is as far away from 0 as is possible. Assuming you bound your latitudes to [-90, 90] and longitudes from (-180, 180], the worst-case will be at the equator for longitude 180.

In binary, 180 requires 8-bits of the 24-bits available, leaving 16 bits after the decimal point. Therefore, the distance between consecutively representable values at this longitude would be 2^-16 deg (approximately 1.526E-5). Multiplying that number (in radians) by the WGS-84 radius of the Earth at the equator (6,378,137 m) yields a worst-case precision of:

2^-16 deg * 6,378,137 m * PI rad / 180 deg = 1.6986 m (5.5728 ft).

The same analysis against lat/longs stored in radians yields the following:

2^-22 rad * 6,378,137 m = 1.5207 m (4.9891 ft)

And finally, if you normalize the latitudes to the range [-1, 1] and the longitudes to the range (-1, 1], then you can achieve the following worst-case precision:

2^-24 * PI rad * 6,378,137 m = 1.1943 m (3.9184 ft)

So storing lat/long in radians buys you around 7 inches of additional accuracy, and storing them in normalized form buys you around 1'8" of additional accuracy, both in the worst-case scenario.

If, when converting between double-precision and single-precision you rounded (instead of truncating), the single-precision value will be within half of the distance between two consecutive values computed above.

Jeff G
  • 4,470
  • 2
  • 41
  • 76
  • You need to use circumference not radius, right? (Deleted my previous comment which had some confusion in it) – HRJ Aug 03 '15 at 14:32
  • 1
    @HRJ We are asking the question, what is the worst-case resolution (in both degrees and radians) between consecutive representable longitudes using a single-precision number? Once we have these numbers (2^-16 and 2^-22, respectively), we convert them to arc lengths using the equatorial (worst-case) circumference. The formula for the full equatorial circumference is 2 * pi * R, and the fraction of the circumference that we are looking for is θ / (2 * pi) (θ in radians). When you multiply these together, you simply get R * θ (θ in radians). – Jeff G Aug 03 '15 at 15:26
  • @HRJ Also see [Radian Definition](https://en.wikipedia.org/wiki/Radian#Definition). The equation *s = Rθ* is actually how the radian is defined. In the equation, *s* is the arc length, *R* is the radius, and *θ* is the subtended angle in radians. – Jeff G Aug 03 '15 at 15:39
  • Thanks for clarifying the circumference v/s radius doubt. However, you seem to be assuming that all 24 bits are available for significand. However one of them is a dedicated sign bit. You probably need to subtract one digit from your answer. – HRJ Aug 04 '15 at 04:40
  • Actually, you do get 24 bits of precision. See [Single Precision](https://en.wikipedia.org/wiki/Floating_point#Internal_representation) in the linked table. The mantissa is stored with an implicit *1* before the decimal point, so you end up with N+1 significant bits, where N is the number of bits in the mantissa. N is 23 for single, so this gets you 24 bits of precision. – Jeff G Aug 04 '15 at 12:33
4

It depends on what you are using your data for. If you use a float it will be ok if the you only need it down to about the meter level of detail. Using the data in graphically applications will cause a jitter effect if the user zooms in to far. For more about jitter and see Precisions, Precisions. Hope this helps.

masebase
  • 4,915
  • 3
  • 24
  • 20
2

Update: Jeff's answer has a better analysis. However...

To improve upon Jeff's answer:

If you divide the actual angle in radians by π, thus encoding the angle in a scale going from 0 to ±1, then it should be possible to use all the digits of the significand (23 bits (24 - 1 sign bit)). The precision would then be:

2^-23 * 6,378,137 m = 0.7603 m (76 cm)

My Old answer:

A 32 bit floating point number can represent a number with about 7.2 decimal digits of precision. This is an approximation because the floating point number is actually in binary, and when converted to decimal, the number of significant digits might vary.

If we take it as 6 decimal digits of precision (to play on the safe side), and if we are storing latitude and longitude in degrees, then we get a precision of about 1/1000th of a degree which is a precision of about 111 meters in the worst case. In the best case, if we get 7 decimal digits of precision, the accuracy would be about 11.1 meters.

It is possible to get a better precision using radians as the unit. In the worst case we get a precision of 10 millionth of a radian which is about 63 meters. In the best case, it would be 1 millionth of a radian which is about 6 meters.

Needless to say, a 64bit floating point number would be extremely precise (about 6 micro meters in the worst case).

HRJ
  • 17,079
  • 11
  • 56
  • 80
  • While your logic is sound, your premise is false. A single precision number gives [from 6 to 9 significant decimal digits](https://en.wikipedia.org/wiki/Single-precision_floating-point_format#IEEE_754_single-precision_binary_floating-point_format:_binary32), depending on the number. The analysis should be done in binary, to alleviate this ambiguity from the result. This false premise is why your conclusion is an order of magnitude off the correct value. See my answer for a more rigorous analysis. – Jeff G Aug 03 '15 at 22:23
  • Your new analysis is incorrect. For Rθ to be the arc length, θ MUST be in radians. However, if you store your longitudes in the range (-1, 1) as you suggest, and can guarantee you are never close enough to the 180th meridian to actually store the value ±1, then you can get better precision. I will update my response to include that analysis. – Jeff G Aug 04 '15 at 12:40
  • After thinking about it, you don't have to meet the requirement of never storing ±1 as a normalized longitude in order to use the full 24-bits of precision in the analysis. This is due to the fact that the next representable longitude in such a system still provides 24-bits of precision. – Jeff G Aug 04 '15 at 12:51
  • @JeffG "θ MUST be in radians", yes, but θ can be computed to be in radians by applying a scale factor. However, I am not sure if the precision will be maintained during multiplication, especially since π is irrational. Can we discuss further in this chat room: http://chat.stackoverflow.com/rooms/85134/lat-long-precision – HRJ Aug 04 '15 at 17:30
  • Don't forget to update your answer per your own comment: "θ can be computed to be in radians by applying a scale factor". That factor is PI, so the worst-case precision presented in your answer needs to be multiplied by PI. I also believe you need to use 2^-24 instead of 2^-23. – Jeff G Aug 05 '15 at 16:30
0

TL;DR: if one-meter resolution is acceptable then a single-precision float storing degrees is acceptable.

This answer is a bit late to the party but I needed a solid answer myself and so hacked out some code to quickly get it. There are of course more elegant ways to do this, but it looks to work. As noted by Jeff, the worst case scenario will be at +/- 180 degrees longitude (ie, the date line).

Per the code below, a single-precision float is accurate to 0.85 meters at the date line using single-precision floats storing degrees. Accuracy increases significantly (to w/in mm) when close to the Prime meridian.

#include <stdio.h>

// per wikipedia, earth's circumference is 40,075.017 KM
#define METERS_PER_DEG  (40075017 / 360.0) 

// worst case scenario is near +/-180.0 (ie, the date line)    
#define LONGITUDE    180.0

int main()                                                                      
{                                                                               
   // subtract very small but increasingly larger values from
   //    180.0 and recast as float until it no longer equals 180.0
   double step = 1.0e-10;   
   int ctr = 1;
   while ((float) LONGITUDE == (float) (LONGITUDE - (double) ctr * step)) {
      ctr++;
   }
   double delta = (double) ctr * step;           
   printf("Longitude %f\n", LONGITUDE);
   printf("delta %f (%d steps)\n", delta, ctr);  
   printf("meters: %f\n", delta * METERS_PER_DEG);
   return 0;                                                                    
} 

Output from this code is

Longitude 180.000000 
delta 0.000008 (76294 steps) 
meters: 0.849301
aquilonis
  • 63
  • 5