uploading multiple files UploadFiles FastAPI

Question

Example

Here's my code:

from typing import  List
from fastapi import FastAPI, File, UploadFile
import asyncio
import concurrent.futures

app = FastAPI()
@app.post("/send_images")
async def update_item(
    files: List[UploadFile] = File(...),
):
    return {"res": len(files)}

And I send requests to this server with (these specific urls are just for example here):

import requests
import os 
import json
import numpy as np
import time
import io
import httpx
import asyncio
from datetime import datetime
import pandas as pd

from random import shuffle
import pandas as pd
import urllib
import io
from PIL import Image
from matplotlib import pyplot as plt
import concurrent.futures

urls = ['https://sun9-63.userapi.com/c638920/v638920705/1a54d/xSREwpakJD4.jpg',
 'https://sun9-28.userapi.com/c854024/v854024084/1160d8/LDMVHYgguAw.jpg',
 'https://sun9-54.userapi.com/c854220/v854220084/111f66/LdcbEpPR6tg.jpg',
 'https://sun9-40.userapi.com/c841420/v841420283/4c8bb/Mii6GSCrmpo.jpg',
 'https://sun6-16.userapi.com/CPQpllJ0KtaArvQKkPsHTZDCupqjRJ_8l07ejA/iyg2hRR_kM4.jpg',
 'https://sun9-1.userapi.com/c638920/v638920705/1a53b/SMta6Bv-k7s.jpg',
 'https://sun9-36.userapi.com/c857332/v857332580/56ad/rJCGKFw03FQ.jpg',
 'https://sun6-14.userapi.com/iPsfmW0ibE8RsMh0k2lUFdRxHZ4Q41yctB7L3A/ajJHY3WN6Xg.jpg',
 'https://sun9-28.userapi.com/c854324/v854324383/1c1dc3/UuFigBF7WDI.jpg',
 'https://sun6-16.userapi.com/UVXVAT-tYudG5_24FMaBWTB9vyW8daSrO2WPFQ/RMjv7JZvowA.jpg']

os.environ['NO_PROXY'] = '127.0.0.1'

async def request_get_4(list_urls):
    async with httpx.AsyncClient() as client:
        r = httpx.post("http://127.0.0.1:8001/send_images", files={f'num_{ind}': el for ind, el in enumerate(list_urls)})
        print(r.text)
        return r

async def request_get_3(url):
    async with httpx.AsyncClient() as client:
        return await client.get(url)
    
from collections import defaultdict

async def main():
    start = datetime.now()
    tasks = [asyncio.create_task(request_get_3(url)) for url in urls[0:10]]
    result = await asyncio.gather(*tasks)
    
    data_to_send = []
    for ind, resp in enumerate(result):
        if resp.status_code == 200:
            image_bytes = io.BytesIO(resp.content)
            image_bytes.seek(0)
            data_to_send.append(image_bytes)
        
    end = datetime.now()
    print(result)
    print(len(data_to_send))

    batch_size = 2
    batch_num = len(data_to_send) // batch_size
    tasks = [asyncio.create_task(request_get_4(data_to_send[i * batch_size: (i+1) * batch_size])) for i in range(batch_num)]
    result = await asyncio.gather(*tasks)
    
    left_data = data_to_send[batch_size*(batch_num):]
    print(len(left_data))
    print(result)

asyncio.run(main())

I am trying to load image which are contained in urls, then form batches of them and send them to FastAPI server. But it doesn't work. I get the following error:

{"detail":[{"loc":["body","files"],"msg":"field required","type":"value_error.missing"}]}

How can I fix my problem and be able to send multiple files through httpx to FastAPI?

My guess is that your endpoint expects a `list` of files, while you are passing a dictionary via the `httpx.post` request. Try something like `files=[f for f in downloaded_files]` , which, BTW, does not seem you are downloading them before sending them — lsabi, Jul 28 '20 at 20:11
and why do you think that I don't download urls? I use httpx.get. — Danil Kononyhin, Jul 29 '20 at 09:53
so, yeah, this is httpx module limitation! right now it can't send more than one file in post request! — Danil Kononyhin, Jul 29 '20 at 11:00
Sorry, I misread and though that list_urls was still the list above. Also, I don't think it's httpx's limitation. https://www.python-httpx.org/advanced/#multipart-file-encoding says that it accepts a dictionary with tuples. Does that work? — lsabi, Jul 29 '20 at 15:12
Please, look up the url I mentioned in my answer [link](https://github.com/encode/httpx/pull/1032) It has all the information. — Danil Kononyhin, Jul 30 '20 at 09:25
Now that I have enough time to read it thoughtfully, I see it's not been released yet. Though the docs said something different.. Sorry, my bad — lsabi, Jul 31 '20 at 07:44

Danil Kononyhin · Accepted Answer · 2021-04-27T07:13:50.163

The problem is that HTTPX 0.13.3 doesn't support multiple file uploading as this arg wants dictionary and dictionary can't have same key values.

We can use this pull request https://github.com/encode/httpx/pull/1032/files to fix this problem (now it can also accept List[Tuple[str, FileTypes]]])!

UPDATE: this problem is solved now!

Update2: Here I show how I use httpx for different requests:

async def request_post_batch(fastapi_url: str, url_content_batch: List[BinaryIO]) -> httpx.Response:
    """
    Send batch to FastAPI server.
    """
    async with httpx.AsyncClient(timeout=httpx.Timeout(100.0)) as client:
        r = await client.post(
            fastapi_url,
            files=[('bytes_image', url_content) for url_content in url_content_batch]
        )
        return r


async def request_post_logs(logstash_url: str, logs: List[Dict]) -> httpx.Response:
    """
    Send logs to logstash
    """
    async with httpx.AsyncClient(timeout=httpx.Timeout(100.0)) as client:
        r = await client.post(
            logstash_url,
            json=logs
        )
        return r

[('images', ('foo.png', open('foo.png', 'rb'), 'image/png')), ('images', ('bar.png', open('bar.png', 'rb'), 'image/png'))] — Ievgen, May 05 '22 at 20:19

score 1 · Answer 2 · answered May 05 '22 at 20:23

This example will pass multiple files under the files field.

async with httpx.AsyncClient(timeout=httpx.Timeout(100.0)) as client:    
     response = await client.post(f"/api/v1/upload-files",
                files=[("files", ("image.png", b"{}", "image/png")), ("files", ("image2.png", b"{}", "image/png"))],
            )

uploading multiple files UploadFiles FastAPI

Example

2 Answers2