Golang http request results in EOF errors when making multiple requests successively

Question

I am trying to debug a very unusual error I am receiving for a simple REST library I wrote.

I am using the standard net/http package to make Get, Post, Put, Delete requests but my tests occasionally fail when I make multiple requests successively. My test looks like this:

func TestGetObject(t *testing.T) {
    firebaseRoot := New(firebase_url)
    body, err := firebaseRoot.Get("1")
    if err != nil {
        t.Errorf("Error: %s", err)
    }
    t.Logf("%q", body)
}  

func TestPushObject(t *testing.T) {
    firebaseRoot := New(firebase_url)
    msg := Message{"testing", "1..2..3"}
    body, err := firebaseRoot.Push("/", msg)
    if err != nil {
        t.Errorf("Error: %s", err)
    }
    t.Logf("%q", body)
}

And I am making the request like this:

// Send HTTP Request, return data
func (f *firebaseRoot) SendRequest(method string, path string, body io.Reader) ([]byte, error) {
url := f.BuildURL(path)

// create a request
req, err := http.NewRequest(method, url, body)
if err != nil {
    return nil, err
}

// send JSON to firebase
resp, err := http.DefaultClient.Do(req)
if err != nil {
    return nil, err
}

if resp.StatusCode != http.StatusOK {
    return nil, fmt.Errorf("Bad HTTP Response: %v", resp.Status)
}

defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
    return nil, err
}

return b, nil
} 

Sometimes it works, but most of the time I get 1 or 2 failures:

--- FAIL: TestGetObject (0.00 seconds)
firebase_test.go:53: Error: Get https://go-firebase-test.firebaseio.com/1.json: EOF
firebase_test.go:55: ""

--- FAIL: TestPushObject (0.00 seconds)
firebase_test.go:63: Error: Post https://go-firebase-test.firebaseio.com/.json: EOF
firebase_test.go:65: ""
FAIL
exit status 1
FAIL    github.com/chourobin/go.firebase    3.422s

The failures happen when I make more than 1 request. If I comment out everything except for the PUT request, the tests consistently pass. Once I include a second test, such as GET, one or the other fails (sometimes both pass).

Answer 1

I experienced this reliably. You need to set Req.Close to true (the defer on resp.Body.Close() syntax used in the examples is not enough). Like this:

client := &http.Client{}
req, err := http.NewRequest(method, url, httpBody)

// NOTE this !!
req.Close = true

req.Header.Set("Content-Type", "application/json")
req.SetBasicAuth("user", "pass")
resp, err := client.Do(req)
if err != nil {
    // whatever
}
defer resp.Body.Close()

response, err = ioutil.ReadAll(resp.Body)
if err != nil {
    // Whatever
}

Answer 2

I agree with the assertion that you shouldn't be hitting outside servers in your unit tests, why not just use the built-in http.Server and serve up the content that you want to test. (There is actually the httptest package to help with this)

I recently ran into this same problem while trying to crawl sitemaps, and this is what I have found so far:

Go by default will send requests with the header Connection: Keep-Alive and persist connections for re-use. The problem that I ran into is that the server is responding with Connection: Keep-Alive in the response header and then immediately closing the connection.

As a little background as to how go implements connections in this case (you can look at the full code in net/http/transport.go). There are two goroutines, one responsible for writing and one responsible for reading (readLoop and writeLoop) In most circumstances readLoop will detect a close on the socket, and close down the connection. The problem here occurs when you initiate another request before the readLoop actually detects the close, and the EOF that it reads get interpreted as an error for that new request rather than a close that occurred prior to the request.

Given that this is the case the reason why sleeping in between requests works is that it gives readLoop time to detect the close on the connection before your new request and shut it down, so that your new request will initiate a new connection. (And the reason why it would intermittently fail is because there is some amount code running between your requests and depending of scheduling of goroutines, sometimes the EOF will be properly handled before your next request, sometimes not). And the req.Close = true, solution works because it prevents the connection from being re-used.

There is a ticket related to this situation: https://code.google.com/p/go/issues/detail?id=4677 (and a dupe ticket that I created that allowed me to reliably reproduce this: https://code.google.com/p/go/issues/detail?id=8122)

Answer 3

I'm going to guess there is no problem with your code. The most likely cause of your problem is because the server is closing the connection. Rate limiting is one possible reason for this.

Your test shouldn't be relying on an external service that's very brittle and not hermetic. Instead you should think about spinning up a test server locally.

Answer 4

My experience with this error was when I entered absolutely empty input for my JSON API!

I should send {} as empty JSON, but I sent so this error happened

Answer 5

I was developing an image download app when this problem occurs.
Tried request.Close=true but not work.
60% requests resulted in a EOF error.

I thought maybe it is an image server problem, not my code.
But php code works fine.

Then I use

var client = &http.Client{
    Transport: &http.Transport{},
}
client.Do(request)

to make request, instead of

http.DefaultClient.Do(request)

problem gone. Not sure why,I guess something with RoundTripper

Answer 6

I encountered this issue while sending an invalid body to a GET request

I could reproduce by making a request similar to below:

var requestBody interface{}
requestData, _ := json.Marshal(requestBody)
payload = strings.NewReader(string(requestData))

req, err := http.NewRequest("GET", url, payload)
...

Answer 7

I faced the same issue, and after investigating for many hours, it turned out that the CDN closed the connection and caused an EOF error on our side. And the reason of the unwanted close was the User-Agent what I set to the request. I used a randomly picked browser User-Agent, which described an 8-years old version of the Chrome. Probably the request was treated as an untrusted one. After updating the User-Agent the issue has been solved.