What is the proper way of sending many requests to Google Maps Geocoding API?

Question

My goal is to geocode a large number of addresses into LatLng objects. I'm only allowed to use Google and I can't use other APIs that may support batch requests. Therefore I have to go with 1 request = 1 address, but I keep getting OVER_QUERY_LIMIT. Not sure how that is possible because, from official docs, I should be able to geocode 50 addresses per second, client-side.

My first thought was to make 50 requests at once, wait for all to return, pause a second, and continue with the next 50 addresses... So the simplified code looked something like this:

const geocoder = new google.maps.Geocoder();
const allAddresses = [ /* 500 addresses */ ];
geocodeAddresses(allAddresses);

function geocodeAddresses(addresses) {
  const last50 = addresses.splice(-50);
  Promise.allSettled(
    last50.map((address) =>
      new Promise((resolve, reject) =>
        geocoder.geocode({
          address
        }, (results, status) => {
          console.log("Geocoded! Response: " + status);
          resolve();
        })))
  ).then(() =>
    setTimeout(() => geocodeAddresses(addresses), 1000)
  );
}

This code successfully returns the first 10 requests at once and then fails on others due to quota limits (why?).

So I decided to go with 1 request at a time with 500ms delay after each result, which is already extremely slow. I'd then repeat OVER_QUERY_LIMIT requests with a 2000ms timeout. The result is very weird:

• first ~200 requests all went ok
• next ~50 requests: ~10 requests repeated
• next ~30 requests: ~15 requests repeated
• next ~10 requests: ~10 requests repeated (each failed before success)

and after ~350 completed requests, each failed 5-6 times and it was really slow. It seems like it may be failing exponentially.

What is the fastest way to geocode all my addresses with Google? Thanks!

Answer 1

Related question:

• OVER_QUERY_LIMIT in Google Maps API v3: How do I pause/delay in Javascript to slow it down?

See:

• Geocoding Addresses Best Practices

This section:

Managing errors and retries

If you get an OVER_QUERY_LIMIT status code as a response, you have exceeded the usage limits for the API. We recommend you try these usage optimization strategies.

• Usage Optimization Strategies

Rate limits
The geocoding service is rate limited to 50 QPS (queries per second), calculated as the sum of client-side and server-side queries.

There is a suggestion in the documentation for the web services: Best Practices Using Google Maps APIs Web Services to use Exponential Backoff:

Exponential Backoff
In rare cases something may go wrong serving your request; you may receive a 4XX or 5XX HTTP response code, or the TCP connection may simply fail somewhere between your client and Google's server. Often it is worthwhile re-trying the request as the followup request may succeed when the original failed. However, it is important not to simply loop repeatedly making requests to Google's servers. This looping behavior can overload the network between your client and Google causing problems for many parties.

A better approach is to retry with increasing delays between attempts. Usually the delay is increased by a multiplicative factor with each attempt, an approach known as Exponential Backoff.