How to deal with large file upload via HTTP without storing the request in temporary files

Question

I want to submit a get request with a large file payload (up to several gbs) to a golang server, but rather than having it store the multi-part body on the file system and then copy the file contents to a buffer and forward it on, I'd like to forward the body of the request to another service as it arrives (without storing anything). Here is a modified (from How can I receive an uploaded file using a Golang net/http server?) non-working example for a post request:

func ReceiveFile(w http.ResponseWriter, r *http.Request) {
    var Buf bytes.Buffer
    file, header, err := r.FormFile("file")
    if err != nil {
        panic(err)
    }
    defer file.Close()
    io.Copy(&Buf, file)
    // stream Buf to grpc server
    Buf.Reset()
    return
}

You should use client-side streaming grpc for large messages. That way you can read the http stream in chunks and send it via grpc without using store-and-forward. — Burak Serdar, Nov 30 '19 at 23:16
@Markus, the problem is this stores the entire file on the server, the exact problem I want to avoid — Dan Jenson, Nov 30 '19 at 23:20
From your question, it looks like you receive a file via http, and you want to send it to another service via grpc. For that, you should use the above code as a starting point, but chunk the data as you read from the http request, and send it to the grpc server using those chunks as a client-side stream. That way, you don't need to store the file. — Burak Serdar, Nov 30 '19 at 23:27
@BurakSerdar, golang stores the file automatically if the request body exceeds 10mb, that's what I'm trying to intercept and avoid. Client side streaming to a grpc service is easy once I get that. Do you know how to receive each multiform part of the data individually and avoid the default aggregation handling? — Dan Jenson, Nov 30 '19 at 23:31
You should edit your question to emphasize that. And for solution: don't call r.FormFile, instead use multipart.NewReader(request.Body), and parse the sections yourself. I can include some code that shows how if you need. — Burak Serdar, Nov 30 '19 at 23:39
@BurakSerdar, if don't mind, I'll mark it as the accepted answer. It will be multipart data content type — Dan Jenson, Nov 30 '19 at 23:48

score 5 · Answer 1 · answered Nov 30 '19 at 23:59

If you're dealing with large multipart data, you should avoid parsing the whole request body. Instead, use the multipart package and read the request body as a stream (error handling is omitted below):

m, p, _:=mime.ParseMediaType(req.Header.Get("Content-Type"))
boundary:=p["boundary"]
reader:=multipart.NewReader(req.Body,boundary)
for  {
  part, err:=reader.NextPart()
  if err==io.EOF {
    // Done reading body
    break
  }
  contentType:=part.Header.Get("Content-Type")
  fname:=part.FileName()
  // part is an io.Reader, deal with it
}

How to deal with large file upload via HTTP without storing the request in temporary files

1 Answers1