1

I want to use HTTPX (within FastAPI, if that matters) to make asynchronous http requests to an outside API and store the responses as individual variables for processing in slightly different ways depending on which URL was fetched. I'm modifying the code from this StackOverflow answer.

import asyncio
import httpx

async def perform_request(client, url):
    response = await client.get(url)
    return response.text

async def gather_tasks(*urls):
    async with httpx.AsyncClient() as client:
        tasks = [perform_request(client, url) for url in urls]
        result = await asyncio.gather(*tasks)
        return result

async def f():
    url1 = "https://api.com/object=562"
    url2 = "https://api.com/object=383"
    url3 = "https://api.com/object=167"
    url4 = "https://api.com/object=884"
    result = await gather_tasks(url1, url2, url3, url4)
    # print(result[0])
    # print(result[1])
    # DO THINGS WITH url2, SOMETHING ELSE WITH url4, ETC.

if __name__ == '__main__':
    asyncio.run(f())

What's the best way to access the individual responses? (If I use result[n] I wouldn't know which response I'm working with.)

And I'm pretty new to httpx and async operations in general so please share if you have any suggestions for how to achieve it in a better way.

noob
  • 328
  • 2
  • 13
  • Why not just returning the `{url: response.text}` – Yagiz Degirmenci Jan 08 '21 at 13:35
  • The urls won't be hardcoded like they are in my example. The url will actually be constructed after being passed in from a FastAPI route. Sorry I wasn't clear enough in my example, I should have mentioned that. – noob Jan 08 '21 at 13:38

1 Answers1

0

Regardless of AsyncIO, I would probably put the logic inside gather_tasks. There you know the response, and you can define all the if else logic you want to proceed with the right path.

In my opinion you have two options:

1 - Process the request right away

In this case f would only initialize the urls and trigger the processing, everything else would happen inside gather_tasks.

2 - "Enrich" the response

In gather_tasks you can understand which kind of operation to do next, and "attach" to the response some sort of code to define it. For example, you could return a dict with two keys: response and operation. This would be the most explicit way of doing this, but you could also use a list or a tuple, you just need to know where the response and the "next step code" is within them.

This is useful if the further processing must happen later instead of right away.

Makes sense?

Sushi2all
  • 475
  • 5
  • 11
  • #1 makes sense but I want to re-use `gather_tasks()` for other sets of URLs later down the line (to be processed differently). Also, `f()` would actually be a FastAPI route. (Perhaps I should have included that information). Regarding your point #2, I had thought about assigning an identifier but just wanted to see if there'd be a more clever way to achieve the same thing. I was hoping that maybe there would be some native way of doing it. – noob Jan 08 '21 at 13:28
  • What would be the problem of reusing `gather_tasks()` for other sets of urls? – Sushi2all Jan 08 '21 at 13:33
  • Other batches of URLs would be processed differently. I was hoping to use `gather_tasks()` just to gather them, and the processing of results happens in the function that calls `await gather_tasks(...)` – noob Jan 08 '21 at 13:35
  • Uhm...don't you still have to define somewhere all the possible options? – Sushi2all Jan 08 '21 at 13:38
  • The next batch of URLs will be created from the json data in the first batch's responses. They will have other operations performed on them, e.g. POST, etc. I'm basically trying to async fetch the URLs but at the same time keep them separate to continue the next step of processing. Not sure if that answers your question but trying to explain as well as I can. – noob Jan 08 '21 at 13:54
  • I don't fully understand, but from what I gather I would say you're best off with option 2. You Simply "tag" the responses so you can pick them up again whenever and you know what to do without inspecting them. – Sushi2all Jan 08 '21 at 14:23