Python async request call inside loop

Question

I'm trying to develop a Python API with FastAPI, and I have to download a series of images from several URLs, in a foor loop.

The problem is that I have to retrieve the images, and I can't do it as it is an async operation inside a normal for loop.

Here is my code:

@app.post("/v1/img_color")
async def img_color(request: Request):
    body = await request.json()
    images = []
    
    for img in body['images']:
        img_id = img['id']
        img_url = img['url']
        r = requests.get(img_url)
        content = await r.content
        dw_image = Image.open(BytesIO(content))
        images.append(dw_image)
    

    return images

But it gives me the following error:

TypeError: object bytes can't be used in 'await' expression

How can I solve this? I've searched about this issue, and found some solutions regarding asyncio but I can't manage to make them work.

Update:

Following the suggestion of deleting the await code from await r.content, I've reached to another error, which says the following:

TypeError('cannot convert dictionary update sequence element #0 to a sequence'), TypeError('vars() argument must have dict attribute')

I don't quite understand this, as I can retrieve perfectly the url from the JSON body of the original POST request...

Traceback

Traceback (most recent call last):
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 388, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\applications.py", line 190, in __call__
    await super().__call__(scope, receive, send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\middleware\errors.py", line 181, in __call__
    raise exc from None
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\middleware\errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\exceptions.py", line 82, in __call__
    raise exc from None
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\routing.py", line 41, in app
    response = await func(request)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\routing.py", line 196, in app
    response_data = await serialize_response(
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\routing.py", line 124, in serialize_response

    return jsonable_encoder(response_content)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\encoders.py", line 102, in jsonable_encoder
    jsonable_encoder(
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\encoders.py", line 140, in jsonable_encoder
    return jsonable_encoder(
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\encoders.py", line 88, in jsonable_encoder
    encoded_value = jsonable_encoder(
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\encoders.py", line 88, in jsonable_encoder
    encoded_value = jsonable_encoder(
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\encoders.py", line 139, in jsonable_encoder
    raise ValueError(errors)
ValueError: [TypeError('cannot convert dictionary update sequence element #0 to a sequence'), TypeError('vars() argument must have __dict__ attribute')]

`requests.get()` is synchronous. There is nothing to `await`. When the call is done, the content bytes are completely available. Remove the `await`. — Tomalak, Dec 05 '20 at 23:08
Does this answer your question? [How could I use requests in asyncio?](https://stackoverflow.com/questions/22190403/how-could-i-use-requests-in-asyncio) — Tomalak, Dec 05 '20 at 23:08
@Tomalak I've deleted the `await` code, but now it gives me a new error (see the update), any idea? Thanks in advance! — Pablo Cañal Suárez, Dec 06 '20 at 12:14
Since I have no idea on what line that error occurs, I can't help you. — Tomalak, Dec 06 '20 at 12:18
@Tomalak It doesn't seem to happen in my `main.py` code, but in the framework instead... — Pablo Cañal Suárez, Dec 06 '20 at 12:24
I'm guessing: You do `return images`, which is a list of `Image` objects. I imagine your framework is configured to try to convert this to JSON, and it *seems* to try to turn your `[Image1, Image2]` into an object `{"key1": "value1", "key2": "value2"}`. That can't work because there aren't any keys in there. You can try to `return enumerate(images)` to see if the error goes away. You'll need to figure out what return output you want from your endpoint, and what you need to return from your handler function to get it. — Tomalak, Dec 06 '20 at 12:41
Maybe you want to do `images.append( (img_id, dw_image) )` instead of `images.append(dw_image)`. Then you would have return value of `[(Id1, Image1), (Id2, Image2), ...]` and the framework can turn that into JSON `'{"Id1": "Image1", "Id2": "Image2", ...}'` easily. — Tomalak, Dec 06 '20 at 12:45
But overall, this error has nothing to do with your question. I'll post my suggestion as an answer, because it solves the immediate issue. What you see now is a different issue entirely. — Tomalak, Dec 06 '20 at 12:47
Many thanks, @Tomalak!! Both suggestions solved my issue. Thank you so much :D — Pablo Cañal Suárez, Dec 06 '20 at 12:56
See modified answer for one last suggestion. Comes without an explanation, consider it an exercise. — Tomalak, Dec 06 '20 at 12:59
Perfect!! Just a minor detail for your edited answer: there is a `.content` missing, so it should be `Image.open(BytesIO(requests.get(img['url']).content))` — Pablo Cañal Suárez, Dec 06 '20 at 13:04

Tomalak · Accepted Answer · 2020-12-06T13:11:19.613

requests.get() is synchronous. There is nothing to await. When the call is done, the content bytes are completely available. Remove the await.

for img in body['images']:
    r = requests.get(img['url'])
    dw_image = Image.open(BytesIO(r.content))
    images.append( (img['id'], dw_image) )

If you're feeling adventurous (or want to learn about dict comprehensions), you can replace your entire loop with one line

@app.post("/v1/img_color")
async def img_color(request: Request):
    body = await request.json()
    return {img['id']: Image.open(BytesIO(requests.get(img['url']).content)) for img in body['images']}

Python async request call inside loop

Update:

Traceback

1 Answers1