0

I am using the requests library to map an array of requests after I get an array from another API request. I am using a loop for the requests but I sure there is a better way to do this because this API request can have 500+ items, so finishing this looping is taking 20+ minutes sometimes.

I tried to use the grequests library and I kept getting recursion complaints. I would love to use an async/map method if possible but after researching apparently the async library is not supported anymore.

self.set_header("Access-Control-Allow-Origin", "*")
response = requests.get("https://hacker-news.firebaseio.com/v0/paststories.json?print=pretty")
data = response.json()
story_list = []

for story in data:
    temp_string = "https://hacker-news.firebaseio.com/v0/item/{}.json?print=pretty".format(story)
    story_data = requests.get(temp_string)
    story_list.append(story_data.json())

There should be a better way to do this looping than the current method because 20+ minutes to get the data is not acceptable. The API response in the original array can return an array of 500+ so the method should be scalable.

Nazim Kerimbekov
  • 4,712
  • 8
  • 34
  • 58
Demetrius
  • 449
  • 1
  • 9
  • 20

2 Answers2

1

requests is synchronous, so your script waits for a response to make a new request. So Maybe you should look into aiohttp and aysnchronous requests.

This could be an example : Is that benchmark reliable - aiohttp vs requests

Loïc
  • 11,804
  • 1
  • 31
  • 49
  • I am using the aiohttp method but I getting a "is not JSON serializable" when I use json.dumps. Is there a fix for this? – Demetrius Jun 02 '19 at 21:15
  • you can get ```response.text()``` I think, see : https://docs.aiohttp.org/en/stable/client_quickstart.html#make-a-request – Loïc Jun 02 '19 at 22:07
1

well, you just need to boost the speed of your io-bond code, there are tones of the solution, please refer to this related answer in stackoverflow: How could I use requests in asyncio?

due to asyncio is too much fundamental, there are a lot of package built base on it, try this package: aiohttp-requests

hope these information can help.

Han.Oliver
  • 525
  • 5
  • 8
  • We discourage answers that are "link-only" because one day these links may become stale and then the answer will become worthless. Please copy from these links the relevant parts into the answer in order to avoid such a situation. – Nir Alfasi Jul 15 '19 at 22:40