2

Looking to transfer a large (200MB) TIFF image to FastAPI backend, the normal fileupload takes 5sec to do it.

What are good practices on having a fast transfer of large files??

Used this approach to have a quicker way, but seems to be only working for smaller files (tested with 1.5 MB TIFF). With the small file I already went from 200 ms to 100ms:

Server side:

@app.post('/upload_tiff_V2')
async def upload_tiff_V2(request: Request):
    start = time.time()
    body = b''
    try:
        filename = request.headers['filename']
        async with aiofiles.open(filename, 'wb') as f:
            async for chunk in request.stream():
                await f.write(chunk)
                body += chunk
        
        arr = np.frombuffer(body, dtype=np.uint8)
        img = cv2.imdecode(arr, cv2.IMREAD_COLOR)

        arr = add_text(img, "AVERNA")

        success, im = cv2.imencode('.tif', arr)
        headers = {'Content-Disposition': 'inline; filename="test.tif"'}
        # record end time
        end = time.time()
        print("The time of execution of the TIFF program is :",
        (end-start) * 10**3, "ms")
    except Exception:
        return {"message": "There was an error uploading the file"}
    return Response(im.tobytes(), headers=headers, media_type='image/tif')

Client Side:

#############################################
## UPLOAD BIG FILE TIFF
#############################################
with open("big_image.tif", "rb") as f:
    data = f.read()
   
url = 'http://192.168.199.130:2123/upload_tiff_V2'
headers = {'filename': 'big_image.tif'}

# record start time
start = time.time()

resp = requests.post(url=url, data=data, headers=headers)

# record end time
end = time.time()

print("The time of execution of upload BIG (200Mb) TIFF image program is :",
      (end-start) * 10**3, "ms")
Chris
  • 18,724
  • 6
  • 46
  • 80
BenEngelen
  • 21
  • 1
  • 1
    It seems like you're timing the operation as well? i.e. loading np, imdecode, imencode, etc.? 200MB/5sec is 40MB/s, about 320mbit/s - which seems decent enough given all the steps of encoding/decode that is involved just for submitting a file through a http service. Make sure you're timing the thing you're actually asking about - attach a profiler to see where you're actually spending your time. – MatsLindh Jan 12 '23 at 14:11
  • it might be worth running the image processing stuff in somewhere else (e.g. using [`loop.run_in_executor()`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor)) so it doesn't block IO unnecessarily. also, try separating out the timing for upload portion, processing and sending the response as each will have their own performance characteristics – Sam Mason Feb 10 '23 at 12:45

0 Answers0