8

I'm using overpy to query the Overpass API, and the nature of the data is such that I have a lot of queries to execute. I've run into the 429 OverpassTooManyRequests exception and I'm trying to play by the rules. I've tried introducing time.sleep methods to space out the requests, but I have no basis for how long the program should wait before continuing.

I found this link which mentions a "Retry-after" header:
How to avoid HTTP error 429 (Too Many Requests) python

Is there a way to access that header in an overpy response? I've been through the docs and the source code, but nothing stood out that would allow me to access that header so I can pause querying until it's acceptable to do so again.

I'm using Python 3.6 and overpy 0.4.

Community
  • 1
  • 1
Chris M.
  • 282
  • 3
  • 13
  • 1
    Instead of spacing out the requests, why not just catch the `OverpassTooManyRequests` exception, wait 10 seconds, then try again? If it's too early, then it'll just wait another 10 seconds. – Peter Apr 26 '17 at 14:03
  • You need to evaluate the results of the /api/status call. BTW: how much is "a lot of queries"? Are you aware of the usage limits? That's 10'000 queries for all of your users per day. Forget about the "Retry-after" header at this time, it's not yet implemented, see https://github.com/drolbr/Overpass-API/issues/351 – mmd Apr 26 '17 at 14:05
  • @mmd about 450 queries, however, this isn't a script that will be making all these queries repeatedly. The queries are made once, the results are added to an HTML map file and the map file is what gets shared. Ideally, you'd make the queries one time for a given data set and never have to again. So the only user making the queries is me. And I am aware of the 10,000/day limit. – Chris M. Apr 26 '17 at 14:08
  • 450 queries doesn't sound too much, unless they're very expensive. Other libraries already implement a logic to evaluate /api/status, here's an example for R, if you don't mind: https://github.com/hrbrmstr/overpass/blob/master/R/overpass_query.r – mmd Apr 26 '17 at 14:14
  • @mmd I'm not familiar with R (I know what it is, but have never used it). Is there a way to evaluate /api/status in Python or OverPy? – Chris M. Apr 26 '17 at 14:16
  • Why not raise an issue with OverPy then? – mmd Apr 26 '17 at 14:17
  • You'll have to forgive me, I'm not used to suggesting...improvements? to a module. Do I just suggest that they add a way to evaluate /api/status on their Github repository? – Chris M. Apr 26 '17 at 14:21
  • Sure, why not? :) – mmd Apr 26 '17 at 14:23

3 Answers3

4

Maybe this isn't quite the answer you're seeking, but I ran into the same issue and fixed it by simply hosting my own OSM database server using docker. Just clone the repo and follow instructions:

https://github.com/mediasuitenz/docker-overpass-api

B.abba
  • 59
  • 4
1

from http://overpass-api.de/command_line.html do check that you do not have a single 'runaway' request that is taking up all the resources.

Rene
  • 41
  • 5
1

After verifying that I don't have runaway queries, I have taken Peter's advice and added a catch for the TooManyRequests exception that waits 30s and tries again. This seems to be working as an immediate solution.

I will also raise an issue with the originators of OverPy to suggest an enhancement to allow evaluating the /api/status, as per mmd's advice.

Chris M.
  • 282
  • 3
  • 13
  • Once you raise an issue with OverPy, can you please also post a link to it here. Thanks. – mmd Apr 28 '17 at 08:21