0

We are trying to implement image uploading through POST requests. We also want to limit the images to ~1,0 MiB. It works fine on smaller images, but anything ~2,5 MiB or larger causes the connection to reset. It also seems to send multiple requests after the first to the same handler.

main.go:

package main

import (
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/", uploadHandler)
    http.ListenAndServe("localhost:8080", nil)
}

func uploadHandler(w http.ResponseWriter, r *http.Request) {
    if r.Method == "POST" {
        postHandler(w, r)
        return
    } else {
        http.ServeFile(w, r, "index.html")
    }
}

func postHandler(w http.ResponseWriter, r *http.Request) {
    // Send an error if the request is larger than 1 MiB
    if r.ContentLength > 1<<20 {
        // if larger than ~2,5 MiB, this will print 2 or more times
        log.Println("File too large")
        // And this error will never arrive, instead a Connection reset
        http.Error(w, "response too large", http.StatusRequestEntityTooLarge)
        return
    }
    return
}

index.html:

<!DOCTYPE html>
<html lang="">
  <head>
    <meta charset="utf-8">
    <title></title>
  </head>
  <body>
    <form method="POST" enctype="multipart/form-data">
      <input type="file" accept="image/*" name="profile-picture"><br>
      <button type="submit" >Upload</button>
  </form>
  </body>
</html>

Output when uploading a ~2,4 MiB file

$ go run main.go
2021/11/23 22:00:14 File too large

It also shows "request too large" in the browser

Output when uploading ~2,5 MiB file

$ go run main.go
2021/11/23 22:03:25 File too large
2021/11/23 22:03:25 File too large

The browser now shows that the connection was reset

urist
  • 23
  • 3
  • 2
    `if r.ContentLength > 1<<20` that's 1,048,576 bytes or 1MiB. `postHandler` considers anything larger than 1M too large. – Schwern Nov 23 '21 at 20:09
  • @Schwern Working as intended. We want to limit the image upload size when uploading, though we don't want to reset the connection when the image uploaded is too large, nor do we want the browser to send multiple requests. Added some clarification to the post though – urist Nov 23 '21 at 20:22
  • What's between the browser and the Go service? Is there a reverse proxy, load balancer, WAF, CDN, etc? – Adrian Nov 23 '21 at 20:26
  • @Adrian There shouldn't be anything between it, it's completely local and self-contained code. – urist Nov 23 '21 at 20:34
  • Then can you show the elided code under "do stuff with image here"? That's likely where it's failing. – Adrian Nov 23 '21 at 20:52
  • Probably because AFAIK this is still the case: browsers don't handle 413 well unless they can finish sending the entire request body. The spec says the server may send a 413 and then may close the connection, but the browsers always say connection reset in that case. https://stackoverflow.com/questions/18367824/how-to-cancel-http-upload-from-data-events – erik258 Nov 23 '21 at 20:54
  • @Adrian Sorry for the misunderstanding. The code I initially posted was quick independent program based on what we were working on, but still had the same problem. I added the comment to avoid confusion on why I wasn't doing anything with the file (while in our main code we actually processed the file). I've removed it now. – urist Nov 23 '21 at 21:43
  • The connection has to be reset if one side doesn't fully read what the other side sends. If the server in this case didn't, it might read leftover bytes from the previous request when it expects to read the next request header. Those bytes could come from buffers in the kernel or the network card or they could have been literally in flight when the server decided to stop reading. – Peter Nov 23 '21 at 22:03
  • 1
    @urist then that's definitely the problem. You're not actually reading the request in your handler, so you're hitting a write timeout on the client. – Adrian Nov 24 '21 at 14:19

1 Answers1

3

The client is trying to send data to the server. The server is not reading the data, it's just looking at the headers and closing the connection. The client is interpreting this as "connection was reset". This is out of your control.

Instead of checking the header, the header can lie, use http.MaxBytesReader to read the actual content, but error if it is too large.

const MAX_UPLOAD_SIZE = 1<<20

func postHandler(w http.ResponseWriter, r *http.Request) {
    // Wrap the body in a reader that will error at MAX_UPLOAD_SIZE
    r.Body = http.MaxBytesReader(w, r.Body, MAX_UPLOAD_SIZE)

    // Read the body as normal. Check for an error.
    if err := r.ParseMultipartForm(MAX_UPLOAD_SIZE); err != nil {
        // We're assuming it errored because the body is too large.
        // There are other reasons it could error, you'll have to
        // look at err to figure that out.
        log.Println("File too large")
        http.Error(w, "Your file is too powerful", http.StatusRequestEntityTooLarge)
        return
    }

    fmt.Fprintf(w, "Upload successful")
}

See How to process file uploads in Go for more detail.

Schwern
  • 153,029
  • 25
  • 195
  • 336
  • This seems to work to an extend and actually suits to our needs. However it still causes a connection reset with a large enough image like [this](https://upload.wikimedia.org/wikipedia/commons/1/17/Tarvasj%C3%B5gi.jpg), so for example, if I wanted to raise the MAX_UPLOAD_SIZE to `20 << 20` for whatever reason I wouldn't be able to upload anything of that size. – urist Nov 23 '21 at 21:23
  • I also thought the `r.ContentLength` could be used as a quick check before any files are even uploaded at all, even though I know that it could be spoofed. I guess you could implement this in the client-side of things – urist Nov 23 '21 at 21:28
  • 1
    I think I figured out why the connection reset from my above comment, you need to actually start using the data (for example with `r.FormFile`), otherwise it will just stop and return, and close the connection once it starts getting annoyed by the constant data being sent by the client. – urist Nov 23 '21 at 22:37