How to save a file received using "streaming_form_data" library to memory instead of saving it to disk?

Question

How should I modify the code below (taken from this post), so that the file is not saved to disk, but saved to memory instead? The code uses the streaming-form-data library for parsing multipart/form-data input chunks.

Unfortunately, the below is currently requiring a path, and I would like to prevent I/O operations like that.

from fastapi import FastAPI, Request, status
from fastapi.exceptions import HTTPException
from streaming_form_data import StreamingFormDataParser
from streaming_form_data.targets import FileTarget, ValueTarget
from streaming_form_data.validators import MaxSizeValidator
import streaming_form_data
from starlette.requests import ClientDisconnect
import os

MAX_FILE_SIZE = 1024 * 1024 * 1024 * 4  # = 4GB
MAX_REQUEST_BODY_SIZE = MAX_FILE_SIZE + 1024

app = FastAPI()

class MaxBodySizeException(Exception):
    def __init__(self, body_len: str):
        self.body_len = body_len

class MaxBodySizeValidator:
    def __init__(self, max_size: int):
        self.body_len = 0
        self.max_size = max_size

    def __call__(self, chunk: bytes):
        self.body_len += len(chunk)
        if self.body_len > self.max_size:
            raise MaxBodySizeException(body_len=self.body_len)
 
@app.post('/upload')
async def upload(request: Request):
    body_validator = MaxBodySizeValidator(MAX_REQUEST_BODY_SIZE)
    filename = request.headers.get('Filename')
    
    if not filename:
        raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, 
            detail='Filename header is missing')
    try:
        filepath = os.path.join('./', os.path.basename(filename)) 
        file_ = FileTarget(filepath, validator=MaxSizeValidator(MAX_FILE_SIZE))
        data = ValueTarget()
        parser = StreamingFormDataParser(headers=request.headers)
        parser.register('file', file_)
        parser.register('data', data)
        
        async for chunk in request.stream():
            body_validator(chunk)
            parser.data_received(chunk)

(...)

Chris · Answer 1 · 2023-01-05T07:04:11.273

The solution is already given in the answer (demonstrating "How to Upload a large File (≥3GB) to FastAPI backend") from which you took the code. If you would like to hold the data in memory instead of writing them to disk, you simply need to use a ValueTarget instead of FileTarget object. For example:

file_ = ValueTarget(validator=MaxSizeValidator(MAX_FILE_SIZE))

To get the file contents afterwards, you can use the .value attribute, for instance:

print(file_.value)

How to save a file received using "streaming_form_data" library to memory instead of saving it to disk?

1 Answers1