Geopy too slow - timeout all the time

Question

I am using geopy to get latitude - longitude pairs for city names. For single queries, this works fine. What I try to do now is iterating through a big list of city names (46.000) and getting geocodes for each city. Afterwards, I run them through a check loop which sorts the city (if it is in the US) in the correct state. My problem is, that I get "GeocoderTimedOut('Service timed out')" all the time, everything is pretty slow and I'm not sure if that is my fault or just geopys nature. Here is the responsible code snippet:

for tweetcount in range(number_of_tweets):

#Get the city name from the tweet
city = data_dict[0]['tweetList'][tweetcount]['user']['location']

#Sort out useless tweets
if(len(city)>3 and not(city is None)): 

    # THE RESPONSIBLE LINE, here the error occurs
    location = geolocator.geocode(city);

    # Here the sorting into the state takes place
    if location is not None:
        for statecount in range(len(data)):
            if point_in_poly(location.longitude, location.latitude, data[statecount]['geometry']):

                state_tweets[statecount] += 1;
                break;

Somehow, this one line throws timeouts at every 2./3. call. City has the form of "Manchester", "New York, New York" or something similar. I already had try - except blocks around everything, but that doesn't really change anything about the problem, so I removed them for now... Any ideas would be great!

unrelated but `if location is not None:` would be a lot more readable — Padraic Cunningham, Jul 19 '15 at 21:48
did you try to increase the timeout? geolocator.geocode(city, timeout=10); — nxpnsv, Jul 20 '15 at 14:24
Even with timeout = 10, I still get the error. If I run the code 5 times, I usually get the error 3-4 time, the other 1-2 times it works just fine, though it takes a bit longer than it should. — Tobias Petri, Jul 21 '15 at 09:27

score 3 · Answer 1 · edited May 23 '17 at 12:16

You will be at the mercy of whatever geolocator service you are using. geopy is just a wrapper around different web-services and hence may fail if the server is busy. I would create a wrapper around the geolocator.geocode call, something like this:

def geocode(city, recursion=0):
    try:
        return geolocator.geocode(city)
    except GeocoderTimedOut as e:
        if recursion > 10:      # max recursions
            raise e

        time.sleep(1) # wait a bit
        # try again
        return geocode(city, recursion=recursion + 1)

This will try again 10 times, after a delay of 1 second. Adjust these numbers to your liking.

If you repeatably ask for the same city, you should consider wrapping it in some kind of memoizing e.g. this decorator. Since you have not posted a runnable code, I have not been able to test this.

score 1 · Answer 2 · answered Feb 14 '19 at 11:25

1

You should change your line :

location = geolocator.geocode(city);

to

location = geolocator.geocode(city,timeout=None);

answered Feb 14 '19 at 11:25

Shalini Baranwal

2,780
4
24
34

Geopy too slow - timeout all the time

2 Answers2

Linked