0

I am trying to upload files asynchronously to an OpenStack object storage using the Python Swift client. It seems the put_object() method from the Swift client is not async, so although I am running it in an async loop, the loop seems to execute sequentially and it is particularly slow. Note this code is running in a FastAPI.

Here is the code:

import asyncio
import time
from io import BytesIO
from PIL import Image
from fastapi import FastAPI

# Skipping the Cloud Storage authentication for conciseness
# [...]

app = FastAPI()

async def async_upload_image(idx, img):
    print("Start upload image:{}".format(idx))

    # Convert image to bytes
    data = BytesIO()
    img.save(data, format='png')
    data = data.getvalue()

    # Upload image on cloud storage
    swift_connection.put_object('dev', image['path'], data)

    print("Finish upload image:{}".format(idx))
    return True

@app.get("/test-async")
async def test_async():
    image = Image.open('./sketch-mountains-input.jpg')
    images = [image, image, image, image ]

    start = time.time()
    futures = [async_upload_image(idx, img) for idx, img in enumerate(images)]
    await asyncio.gather(*futures)
    end = time.time()
    print('It took {} seconds to finish execution'.format(round(end-start)))

    return True

The logs return something like this which seems particularly slow:

Start upload image:0
Finish upload image:0
Start upload image:1
Finish upload image:1
Start upload image:2
Finish upload image:2
Start upload image:3
Finish upload image:3
It took 10 seconds to finish execution

I even tried to run that same code synchronously and it was faster:

Start upload image:0
Finish upload image:0
Start upload image:1
Finish upload image:1
Start upload image:2
Finish upload image:2
Start upload image:3
Finish upload image:3
It took 6 seconds to finish execution

Is there a solution to do an async put_object() / upload to OpenStack?

Alexis.Rolland
  • 5,724
  • 6
  • 50
  • 77
  • Does this answer your question? [FastAPI runs api-calls in serial instead of parallel fashion](https://stackoverflow.com/questions/71516140/fastapi-runs-api-calls-in-serial-instead-of-parallel-fashion) – Chris Jan 27 '23 at 16:19
  • Thank you @Chris, I went through that FastAPI documentation and it's clear. I guess I will just stick to `def` rather than `async def`, but it would be nice if someone had a solution to do an async `put_object()` kind of thing... I assume it could optimize the upload even further. I'll rephrase the question in that sense. – Alexis.Rolland Jan 28 '23 at 04:46

0 Answers0