I am using multi threading method with futures.ThreadPoolExecutor for getting results from 1000 of urls in my Django application, this is working well but one clarification i want is will it be correct to process the data then and there once we receive 1st response and about to recieve 2nd response and so on ?
Saving whole json response from 100 URLS and then processing one by one also looks in-correct. Can some one suggest what would be best way of doing it ?
My Question is related to below discussion.
A very simple multithreading parallel URL fetching (without queue)
