Getting too many open files during load test with gin gonic

Question

To do a simulated load test have setup a very basic REST api in Go gin-gonic framework with the below code and after around 1000+ requests getting error

http: Accept error: accept tcp [::]:8123: accept4: too many open files; retrying in 1s

func main() {
    gin.SetMode(gin.DebugMode)
    router := gin.Default()

    router.GET("/dummyRequest", func(c *gin.Context) {
        c.Data(http.StatusOK, "application/json; charset=utf-8", []byte(`{"name": "test", "age": 99}`))
    })

    router.Run(":8123")
}

Based on this question here I understand that this can be fixed by executing the ulimit command, but this will only be delaying the issue.

When I execute the netstat -tc command I see that new connections are made and continue to be in ESTABLISHED state long after the request is served.

As expected when I checked the /proc/$PID/limits as expected the below entry is seen:

Max open files            1024                 4096                 files

Please help me what other options I could try to fix this the right way.

I am using curl requests to send requests to test out the above.

Increasing `ulimit -n` to the 10K-60K range is an extremely routine setup for this sort of test, and for running in production. — David Maze, Sep 22 '18 at 12:24

score 3 · Answer 1 · answered Oct 07 '18 at 17:07

Go’s http package doesn’t specify request timeouts by default. You should always include a timeout in your service. What if a client doesn't close their session? Your process will keep alive old sessions hitting ulimits. A bad actor could intentionally open thousands of sessions, DOSing your server.

Try something like this:

srv := &http.Server{
    Addr:           ":8123",
    Handler:        router,
    ReadTimeout:    10 * time.Second,
    WriteTimeout:   10 * time.Second,
}
srv.ListenAndServe()

You can validate before and after by monitoring files opened by your process:

lsof -p [PID_ID]

For reference: https://github.com/gin-gonic/gin#custom-http-configuration

Getting too many open files during load test with gin gonic

1 Answers1