I have the task to make a reverse proxy app which extracts some specific info from request/response body and send it into logstash. So i decided to do some load testing first. I wrote very simple backend web server and basic proxy.
Backend server code:
package main
import "net/http"
func main() {
http.HandleFunc("/", handler)
http.ListenAndServe(":3001", nil)
}
func handler(w http.ResponseWriter, r *http.Request) {
r.Body.Close()
w.Write([]byte("test"))
}
Reverse proxy server code:
package main
import (
"net/http"
"net/http/httputil"
"net/url"
)
var (
proxy *httputil.ReverseProxy
)
func init() {
url, _ := url.Parse("http://localhost:3001")
proxy = httputil.NewSingleHostReverseProxy(url)
proxy.Transport = &mimTransport{}
}
func main() {
http.ListenAndServe(":3000", proxy)
}
func handler(w http.ResponseWriter, r *http.Request) {
r.Host = "localhost:3001"
proxy.ServeHTTP(w, r)
}
type mimTransport struct {
}
func (t *mimTransport) RoundTrip(request *http.Request) (*http.Response, error) {
response, err := http.DefaultTransport.RoundTrip(request)
if response != nil {
defer request.Body.Close()
}
// info extraction will be here
return response, err
}
As you can see backend server listening localhost:3001
and proxy listening localhost:3000
I'am using apache benchmark for load testing.
When i running $ ab -c 20 -n 20000 -s 10 http://127.0.0.1:3000/
around 12k-14k requests proxy start writing errors like:
2016/12/23 13:02:50 http: proxy error: read tcp [::1]:58330->[::1]:3001: read: connection reset by peer
2016/12/23 13:02:50 http: proxy error: read tcp [::1]:58461->[::1]:3001: read: connection reset by peer
2016/12/23 13:02:50 http: proxy error: read tcp [::1]:58977->[::1]:3001: read: connection reset by peer
2016/12/23 13:02:50 http: proxy error: read tcp [::1]:59459->[::1]:3001: read: connection reset by peer
...
and soon after that apache bench dies by timeout. Some bench results:
Benchmarking 127.0.0.1 (be patient)
Completed 2000 requests
Completed 4000 requests
Completed 6000 requests
Completed 8000 requests
Completed 10000 requests
^C
Server Software:
Server Hostname: 127.0.0.1
Server Port: 3000
Document Path: /
Document Length: 4 bytes
Concurrency Level: 20
Time taken for tests: 4.868 seconds
Complete requests: 11514
Failed requests: 22
(Connect: 0, Receive: 0, Length: 22, Exceptions: 0)
Non-2xx responses: 22
Total transferred: 1381790 bytes
HTML transferred: 45968 bytes
Requests per second: 2365.28 [#/sec] (mean)
Time per request: 8.456 [ms] (mean)
Time per request: 0.423 [ms] (mean, across all concurrent requests)
Transfer rate: 277.20 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 4.2 1 111
Processing: 0 3 4.8 3 113
Waiting: 0 3 4.7 3 113
Total: 0 4 6.4 4 116
Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 5
80% 5
90% 5
95% 6
98% 6
99% 7
100% 116 (longest request)
There is more, if i start ab
again right after the end of first test, i'll receive something like:
Benchmarking 127.0.0.1 (be patient)
apr_pollset_poll: The timeout specified has expired (70007)
Total of 52 requests completed
(Maybe some connection from previous test still alive)
Internet says there is some DoS protection mechanism in Go, and apparently it is cause of reseted connections. So my questions is:
- Why it reset connections (read buffer is full, timeout is happens, too many connections or maybe my code is broken)
- Can i tweak this mechanism? For example extend buffer of change max timeout.
- Some hints how correctly handle this type of errors.
Thanks.