Avoiding strain on APIs when making a aggregating website

Question

My plan is to fetch a bunch of different API's results and then nicely aggregate the results for users but I'm having trouble figuring out how to handle as many users as possible while not breaking any rules when using APIs I have access to. The plan was to use Flask/Python but the APIs have all a bit different limitations and I'd really appreciate any pointers how to handle that (across multiple workers).

API usage requirements (I might want to add more in the future):
API #1: 2 req/s
API #2: 10k req/mo
API #3: 100k req/mo

I thought about making a central controller that returns if the worker may use the APIs it needs but that sounds slow, would it?

Downvoters could you kindly explain your votes? – Avamander Jul 11 '18 at 20:40 — Avamander, Jul 11 '18 at 20:40

score 1 · Accepted Answer · answered Jul 11 '18 at 10:08

1

For these types of problems are good distributed task queues. On Python, popular task queue is Celery.

There you can setup periodic tasks or limiting task rate: How to put rate limit on a celery queue?.

answered Jul 11 '18 at 10:08

Andrej Kesely

168,389
15
48
91

But how can a celery queue be shared across workers (when using uwsgi)? – Avamander Jul 11 '18 at 11:20
1

@Avamander there's tutorial using flask and celery: http://flask.pocoo.org/docs/0.12/patterns/celery/ – Andrej Kesely Jul 11 '18 at 11:27
Thanks, will look into it. – Avamander Jul 11 '18 at 11:34

Avoiding strain on APIs when making a aggregating website

1 Answers1