I am trying implement a web service which would need to make 10000+ request to facebook's graph api ,
should this be done with multithreading like this -What is the fastest way to send 100,000 HTTP requests in Python?
Or
With Asynchronous Programming with asyncio like this- http://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html
Which would be the better option and Why??
First and foremost, 10000+ requests seems a bit excessive especially to a single endpoint. If you can settle for a lower number and don't mind pooling simultaneous requests, Celery should be a good solution in conjunction with Django. There's also a utility called Flower which helps manage Celery. Another good option is Crochet (a Twisted project) I've used it with Flask before and it worked well. Your real limiting factor will be how many open file descriptors (ie. open sockets) your OS supports and how much memory you have.
As far as async vs threads, this is a rabbit hole that is highly bias depending on who you ask. Most will opt for a threaded approach as it's a bit easier to get up in running (initially) and for whatever reason, threads make more sense to the human mind (initially). Async works great if you have functions that yield results rather quickly, but slow down the entire app if a function takes too long or aren't developed properly. With careful designing, async and threads can both yield fast results in their own right. However, I'm of the mindset that you really need both async and threads to balance out the cons of each other. The best design pattern I've found is to use async (Twisted) for the bulk of the work and for the certain tasks that take longer than and block the event loop, I perform those in a thread (also managed by Twisted). As a final note, any sort of async or threaded application needs extreme care and patience when designing because, just like in the real world, doing multiple things at the same time can cause issues.
