1

I have a problem launching multiple go routines over 90000 to do http requests via POST to the same server, both the server and the client are locally, some of the requests are executed successfully and some of them giving me this response read: connection reset by peer

Please notice I am posting a small amount of data in the provided example but actually I am sending a huge amount of data using protobuf.

This is the server https://play.golang.org/p/r1-rYNuAos

package main

import (
    "net/http"
    "log"
    "encoding/json"
)

var port string

type problem struct{
}

func main() {
    p := &problem{}
    p.server(":9090")
}

func (self *problem)server(port string) {
    s := &http.Server{
        Addr:           port,
        Handler:        self,
    }
    log.Println("Server started")
    // Should be last line as it is a blocking.
    log.Fatal(s.ListenAndServe())
}

func (self *problem) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusCreated)
    if err := json.NewEncoder(w).Encode(map[string]interface{}{"created": true}); err != nil {
        log.Fatal(err.Error())
    }
}

And this is the client https://play.golang.org/p/Xx5qQoqrYm

package main

import (
    "net/http"
    "bytes"
    "io/ioutil"
    "log"
    "fmt"
)

type problem struct{
}

func main() {
    p := &problem{}
    p.client(":9090")
}

func (self *problem)client(port string) {
    var k int

    for k=0;k<90000;k++ {

        go func(){

            nativeRequest, err := http.NewRequest(
                "POST",
                "http://127.0.0.1" + port + "/",
                bytes.NewBuffer([]byte(`testing`)),
            )

            nativeRequest.Close = true

            if err != nil {
                fmt.Println(err.Error())
            }

            client := &http.Transport{
            }
            nativeResponse, err := client.RoundTrip(nativeRequest)

            if err != nil {
                log.Println(err.Error())
            }

            if nativeResponse != nil {
                defer nativeResponse.Body.Close()

                if err != nil {
                        fmt.Println(err.Error())
                }

                body, err := ioutil.ReadAll(nativeResponse.Body)

                if err != nil {
                        fmt.Println(err.Error())
                }

                fmt.Println(string(body))
            }
        }()
    }
}
cnicutar
  • 178,505
  • 25
  • 365
  • 392
Ahmed Kamal
  • 126
  • 1
  • 2
  • 10
  • 2
    Does the same issue occur with 100 goroutines rather than over 90000? It could be a restriction with your operating system such as through `ulimit`, or perhaps packets are being dropped due to there being too much traffic? – 1lann Aug 12 '16 at 06:56
  • 3
    Using a single IP and no other configuration, you only have 16k-28k ephemeral ports (depending on the system defaults) for the client, so you can't connect 90k clients concurrently to begin with. Creating a new transport for each connection and then discarding it losing access the idle connections, and they will remain open using up resources -- the http.Client and http.Transport are meant for concurrent use. When trying to test the limits of the system, you can't just pick a "really big number" and then ask why it doesn't work, you need to start from a known working configuration and ramp up. – JimB Aug 12 '16 at 12:44
  • I think [this answer](http://stackoverflow.com/questions/17714494/golang-http-request-results-in-eof-errors-when-making-multiple-requests-successi/23963271#23963271) is much close to the reason. – Ahmed Kamal Aug 13 '16 at 22:24

0 Answers0