2

I have written a Diffbot API. It has 10,000 calls and 1 call per second. What should I do when the limit is exceeded?

Swader
  • 11,387
  • 14
  • 50
  • 84
blackmamba
  • 1,952
  • 11
  • 34
  • 59

3 Answers3

1

You should make sure requests are only made once per second. For front facing user applications, you may want to queue requests so the 1 call/second limit isn't reached.

When the limit of 10,000 calls per month is reached, you should consider paying Diffbot for API usage. The entry level pricing plan is $299 for 250,000 calls, which is quite high. If you only plan on making much less than 250,000 calls per month, you may just want to register for another free developer token.

Jeff Tsui
  • 1,266
  • 12
  • 20
1

When you say you "wrote an API", does that mean you wrote a Python library for accessing it? If so, when the limit is exceeded, the Diffbot API will return a 429 error, telling you you've exceeded the quota, as per errors.

Once this happens, you have several choices, depending on your needs and wants.

  1. If you registered for several free trial tokens, have your Python script consume a list of them on initialization. Once the limit is hit and an error 429 is detected, make sure your script discards the token it used up until that point, and moves onto the next one. Obviously, this is akin to gaming the system and I advise against it.
  2. Log an error and let the end user know
  3. Start paying for usage. If you really crawl more than 10k sites per seven days, you should consider contributing to the service that's making it happen by buying a commercial token.
Swader
  • 11,387
  • 14
  • 50
  • 84
0

There are few more things which can be done. However, it depends on whether you want to continue using it to ensure results for your application and just wish to find an optimum way of doing it or you want to stop(avoid making any further requests).

  • You can call the account API after every certain no. of calls and check if the credits consumed have crossed the threshold and then restrict any further calls. This can be implemented by a flag variable whose value you can flip when the limit is exceeded so that no calls are made further.

  • You can also create child tokens out of the parent token from here and then put some calculated limit on that token. After that, once the limit is exceeded, you can decide upon if you want to go ahead with more calls by again changing the token, or else it will anyway, restrict any further calls being made.

    To implement this, you need to take the token from an external source in the code and then call a small job that can replace its value when the first time limit is exceeded. So the next time when API call will be made to Diffbot, the token value will be changed and you will keep getting responses.

  • Another technique to take action in advance can be to run cron jobs/azure functions daily or weekly which can publish messages using a webhook about the credits consumed for the current bill cycle so that we are aware of the status.

    This is something which I have been doing for a long time for my use case and that way I can take the necessary steps in time.

Naman Sinha
  • 72
  • 5