How to send multiple images from FastAPI backend to JavaScript frontend using StreamingResponse?

Question

I have a FastAPI endpoint /image/ocr that accepts, processes and returns multiple images as a StreamingResponse:

async def streamer(images):
    for image in images:
        # Bytes
        image_bytes = await image.read()

        # PIL PNG
        image_data = Image.open(BytesIO(image_bytes))

        # Process
        # image_processed = ...
    
        # Convert from PIL PNG to JPEG Bytes
        image_ocr = BytesIO()

        image_ocr.save(image_processed, format='JPEG')

        yield image_ocr.getvalue()

@router.post('/ocr', summary='Process images')
async def ocr(images: List[UploadFile]):
    return StreamingResponse(
        streamer(images),
        status_code=status.HTTP_200_OK,
        media_type='image/jpeg'
    )

In my React application I send multiple images to the /image/ocr endpoint using Axios configured to arraybuffer, and convert the returned images to Base64 using btoa, String.fromCharCode and Unit8Array:

const imageSubmit = async data => {
    const formData = new FormData()
    
    // A `useRef` that stores the uploaded images
    data?.current?.files?.successful.map(image =>
        formData.append('images', image.data)
    )

    try {
        const response = await request.post(
            '/image/ocr',
            formData,
            {responseType: 'arraybuffer'}
        )

        const base64String = btoa(
            String.fromCharCode(
                ...new Uint8Array(response.data)
            )
        )

        const contentType = response.headers['content-type']

        const fullBase64String = `data:${contentType};base64,${base64String}`

        return fullBase64String
    } catch (error) {
        // Error log
    }
}

The problem is that I'm sending multiple images, for example, this image and this image, that are being accepted, processed and returned by the endpoint /image/ocr, but when I convert the response.data using the method in the last code snippet I only get one Base64 image.

I took a look at the raw response to see if I could iterate over the ArrayBuffer that I get through response.data:

Array Buffer object

But it didn't work with a for(const image of response.data) {}. I saw that ArrayBuffer has a slice method, but I'm not sure if it means that both images are in the ArrayBuffer and I need to slice and then convert, which is strange since at least one image is being converted to Base64 as of now, or convert it to another type since I saw that it can be done in some different ways here, here and here.

If the slice option is the right one I'm not sure how to select the start and end since the values in the ArrayBuffer are just random numbers and symbols.

Any idea on how to get both images in Base64 on my React application?

Answer 1

After hours of research I gave up on Axios since the documentation says that it "makes XMLHttpRequests from the browser", meaning that it doesn't support stream.

Reading more about the Streams API and watching a Google's series on Youtube, I started using Fetch API, as it supports streaming for both writing (16:26) and reading (6:24).

By replacing the try block in my question with this:

const response = await fetch('http://localhost:4000/image/ocr', {
    method: 'POST',
    headers: {
        Authorization: ''
    },
    body: formData
})

const reader = response.body.getReader()

while (true) {
    const { value, done } = await reader.read()

    // Base64
    const base64 = `data:${response.headers['content-type']};base64,${btoa(String.fromCharCode(...new Uint8Array(value)))}`

    console.log(base64)

    if (done) break
}

I'm able to use the asynchronous generator along with the StreamingResponse and get multiple images in my React application.

Its worth to dive deeper in the stream API since it can also work as a Websocket without having to write a specific back-end that deals with sockets