Python FastAPI server - Streaming in-memory images to HTML5

Question

Goal

I'm trying to stream in-memory images from a Python backend written in FastAPI to a HTML (Vue3) frontend. However, I cannot get the video element to display the stream of images like a video.

Minimal example attempt

As a minimal example, I create 2 images in memory (red and blue) on the server side, which should be displayed on the frontend alternating each second.

I created the following folder and HTML file: templates/index.html:

<!DOCTYPE html>
<html>
<head>
    <title>Streaming JPEG</title>
</head>
<body>
    <h1>Streaming JPEG</h1>
    <p>Your IP address is {{ request.client.host }}.</p>
    <video src="http://127.0.0.1:8000/video-stream" width="256" height="256" autoplay></video> <!-- 0.0.0.0 -->
    <p>End video</p>
</body>
</html>

And I created the following backend file main.py:

import io
from fastapi import FastAPI, Request
from fastapi.responses import HTMLResponse, StreamingResponse
from fastapi.templating import Jinja2Templates
from PIL import Image
import asyncio

app = FastAPI()

templates = Jinja2Templates(directory="templates")


@app.get("/", response_class=HTMLResponse)
async def read_item(request: Request):
    return templates.TemplateResponse("index.html", {"request": request})


@app.get("/video-stream")
async def video_stream():
    # create a red and a blue image
    imgs = [
        Image.new('RGB', (256, 256), color='red'),
        Image.new('RGB', (256, 256), color='blue')  # .tobytes()
    ]
    img_bytes = list()
    for img in imgs:
        # Save the image to a BytesIO object as JPEG format
        with io.BytesIO() as output:
            img.save(output, format="JPEG")
            img_bytes.append(output.getvalue())

    # generator function that will generate JPEG-encoded frames
    async def generate_frames():
        yield b'--frame\r\n'
        while True:
            for idx in range(len(img_bytes)):
                print(f"idx: {idx}")
                yield b'Content-Type: image/jpeg\r\n\r\n' + img_bytes[idx] + b'\r\n--frame\r\n'
                await asyncio.sleep(1)

    return StreamingResponse(generate_frames(), media_type="multipart/x-mixed-replace;boundary=frame")


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="127.0.0.1", port=8000)  # "0.0.0.0"

Which can be started with one of the following commands:

uvicorn main:app --reload
python main.py

Explanation

While searching online it seemed I had to append Content-Type to a byte version of an image before sending it through StreamingResponse, however it doesn't seem to work for me.

Problems

When navigating to http://127.0.0.1:8000/ I see the HTML template with a white area where the video is supposed to play alternating red / blue images.

Looking in my Firefox console (also tried with Chromium), I see the following 2 messages:

HTTP “Content-Type” of “multipart/x-mixed-replace” is not supported. Load of media resource http://127.0.0.1:8000/video-stream failed.
Cannot play media. No decoders for requested formats: multipart/x-mixed-replace

Which seems to indicate it's not properly encoded? Or am I doing something else wrong? Any help is appreciated!

Python FastAPI server - Streaming in-memory images to HTML5 element

Goal

Minimal example attempt

Explanation

Problems

0 Answers0

Python FastAPI server - Streaming in-memory images to HTML5