Gmail API limitations for getting mails

Question

As I understand currently google's API provides 10 requests per second to its API (from their docs), and it looks to be far from enough for comfortable work with mail. I need to get all emails' headers (or at least senders and recipients). Is there anything better than waiting numberOfMails / 10 seconds?

I'm currently accessing API from client side JavaScript application, I'm thinking of distributing API calls for the same user over a number of machines/applications, but's still unclear if their limits apply to gmail user or registered application.

Anyway, hope to get some ideas what to do with it, with current quota it's completely not usable.

The page you linked to states pretty clearly the usage limits: 10,000,000,000 quota units per day for *all* application requests, and 10 requests per second *per gmail user*. — Frxstrem, Jul 05 '14 at 17:47
@Frxstrem is there something better then getting emails one by one then? — Artem Volkhin, Jul 05 '14 at 17:51
@ArtemVolkhin I answered this question. Maybe you have the same question: http://stackoverflow.com/questions/24562981/bulk-fetching-emails-in-the-new-gmail-api/24586740#24586740 — gitter, Jul 05 '14 at 19:48
@gitter it works (with limitation of 1000 request per batch -- https://developers.google.com/gmail/api/guides/batch) add it as an answer and I'll accept it — Artem Volkhin, Jul 05 '14 at 20:28

score 5 · Answer 1 · answered Jul 07 '14 at 22:29

5

The 10 requests/second/user limit you quote isn't enforced at one-second granularity but a longer moving window. You should be able to exceed that limit (i.e. significantly) for some number of seconds before you get pushback. It's intentionally written to allow short-term bursts for a user, etc.

Batching will help with throughput but will not allow you to exceed this limit over a long window (100 requests in batch still counts as 100 requests). I would not send more than 50 or 100 requests in a batch or you will definitely notice some of them getting throttled (429).

And yes, the project-wide limits are significantly more generous than 10 requests/second.

answered Jul 07 '14 at 22:29

Eric D

6,901
1
15
26

Thanks! 0) And each batch can include up to 1000 messages. Why 50 or 100 and what is 429? I also noticed that each additional hundred of requests in batch slows done response time by approximately 1 second. 1) Do you have any estimations on how many messages one can get using batches? 2) Does it make sense to send several batches in parallel, not waiting for the first sent request to finish? – Artem Volkhin Jul 08 '14 at 09:15
1

(0) Didn't say you couldn't send 1000, it's allowed, said I would not (recommend it). "429" is the throttled HTTP response which will happen when you exceed limits and your client is expected to implement some type of exponential backoff. (1) Yes, 50 or 100 at most is ideal presently. (2) No, you'll likely just hit 429 errors still. What are you trying to do that sending 50 or 100 requests at a time is insufficient? – Eric D Jul 08 '14 at 16:00
I want to make an app for analytics. For example, I want to get all senders to show top 10 most frequent senders. I can't find any better solution rather than downloading all mails and parsing their headers. Looks like this API is a wrong tool for my task, IMAP will work faster, how do you think? – Artem Volkhin Jul 08 '14 at 19:21
@EricDeFriez For whatever it is worth I too am finding the 50-100 requests to be an unfortunate limitation. If your program processes every email in a user's inbox and the user steps away for a few days before opening your program again, a non-trivial processing time results simply due to the limitation of down throttling the requests. – Dan Klos Jul 23 '15 at 20:59

gitter · Accepted Answer · 2018-07-16T14:27:31.807

You can use batch requests to send multiple requests together. I spent a whole day this week figuring this out. The java code goes like this:

BatchRequest batchRequest = service.batch();
//callback function. (Can also define different callbacks for each request, as required)
JsonBatchCallback<Thread> callback = new JsonBatchCallback<Thread>() {

    @Override
    public void onSuccess(Thread t, HttpHeaders responseHeaders)
        throws IOException {
        System.out.println(t.getMessages().get(0).getPayload().getBody().getData());
    }

    @Override
    public void onFailure(GoogleJsonError e, HttpHeaders responseHeaders)
        throws IOException {

    }
};

// queuing requests on the batch request
for (Thread thread : threads) {
    service.users().threads().get("me", threads.getId()).queue(batchRequest, callback);
 }


batchRequest.execute();

Added by question's author: for those who also have this problem: https://developers.google.com/gmail/api/guides/batch and https://developers.google.com/api-client-library/javascript/features/rpcbatch. Although RpcBatch is deprecated it's working now and with limitation of 1000 request per batch.

Gmail API limitations for getting mails

2 Answers2

Linked