2

I have the task to make a reverse proxy app which extracts some specific info from request/response body and send it into logstash. So i decided to do some load testing first. I wrote very simple backend web server and basic proxy.

Backend server code:

package main

import "net/http"

func main() {
    http.HandleFunc("/", handler)
    http.ListenAndServe(":3001", nil)
}

func handler(w http.ResponseWriter, r *http.Request) {
    r.Body.Close()
    w.Write([]byte("test"))
}

Reverse proxy server code:

package main

import (
    "net/http"
    "net/http/httputil"
    "net/url"
)

var (
    proxy *httputil.ReverseProxy
)

func init() {
    url, _ := url.Parse("http://localhost:3001")
    proxy = httputil.NewSingleHostReverseProxy(url)
    proxy.Transport = &mimTransport{}
}

func main() {
    http.ListenAndServe(":3000", proxy)
}

func handler(w http.ResponseWriter, r *http.Request) {
    r.Host = "localhost:3001"
    proxy.ServeHTTP(w, r)
}

type mimTransport struct {
}

func (t *mimTransport) RoundTrip(request *http.Request) (*http.Response, error) {
    response, err := http.DefaultTransport.RoundTrip(request)
    if response != nil {
        defer request.Body.Close()
    }

    // info extraction will be here

    return response, err
}

As you can see backend server listening localhost:3001 and proxy listening localhost:3000

I'am using apache benchmark for load testing.

When i running $ ab -c 20 -n 20000 -s 10 http://127.0.0.1:3000/ around 12k-14k requests proxy start writing errors like:

2016/12/23 13:02:50 http: proxy error: read tcp [::1]:58330->[::1]:3001: read: connection reset by peer
2016/12/23 13:02:50 http: proxy error: read tcp [::1]:58461->[::1]:3001: read: connection reset by peer
2016/12/23 13:02:50 http: proxy error: read tcp [::1]:58977->[::1]:3001: read: connection reset by peer
2016/12/23 13:02:50 http: proxy error: read tcp [::1]:59459->[::1]:3001: read: connection reset by peer
...

and soon after that apache bench dies by timeout. Some bench results:

Benchmarking 127.0.0.1 (be patient)
Completed 2000 requests
Completed 4000 requests
Completed 6000 requests
Completed 8000 requests
Completed 10000 requests
^C

Server Software:        
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /
Document Length:        4 bytes

Concurrency Level:      20
Time taken for tests:   4.868 seconds
Complete requests:      11514
Failed requests:        22
   (Connect: 0, Receive: 0, Length: 22, Exceptions: 0)
Non-2xx responses:      22
Total transferred:      1381790 bytes
HTML transferred:       45968 bytes
Requests per second:    2365.28 [#/sec] (mean)
Time per request:       8.456 [ms] (mean)
Time per request:       0.423 [ms] (mean, across all concurrent requests)
Transfer rate:          277.20 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   4.2      1     111
Processing:     0    3   4.8      3     113
Waiting:        0    3   4.7      3     113
Total:          0    4   6.4      4     116

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      5
  80%      5
  90%      5
  95%      6
  98%      6
  99%      7
 100%    116 (longest request)

There is more, if i start ab again right after the end of first test, i'll receive something like:

Benchmarking 127.0.0.1 (be patient)
apr_pollset_poll: The timeout specified has expired (70007)
Total of 52 requests completed

(Maybe some connection from previous test still alive)

Internet says there is some DoS protection mechanism in Go, and apparently it is cause of reseted connections. So my questions is:

  1. Why it reset connections (read buffer is full, timeout is happens, too many connections or maybe my code is broken)
  2. Can i tweak this mechanism? For example extend buffer of change max timeout.
  3. Some hints how correctly handle this type of errors.

Thanks.

  • `ab` is using http/1.0 without keepalive. Don't use `ab`. – JimB Dec 23 '16 at 11:28
  • ab is not your best choice. Have a look at wrk which is like ab but better in so many ways or vegeta (https://github.com/tsenart/vegeta) – StevieB Dec 23 '16 at 11:57
  • Maybe it not best, and i definitely will try other solutions (thanks for it), but i can't guarantee that this problem won't occur in production. So i want to now how to fix/debug it. – Mark Belotskiy Dec 23 '16 at 15:03
  • @MarkBelotskiy: see https://stackoverflow.com/questions/37774624/go-http-get-concurrency-and-connection-reset-by-peer/37813844#37813844, https://stackoverflow.com/questions/30352725/why-is-my-hello-world-go-server-getting-crushed-by-apachebench/30357879#30357879 and https://stackoverflow.com/questions/39813587/go-client-program-generates-a-lot-a-sockets-in-time-wait-state/39834253#39834253 – JimB Dec 23 '16 at 16:01
  • @JimB So it either OS limit or http.Transport limit... thanks for hints. I'll try both and report. – Mark Belotskiy Dec 24 '16 at 09:22
  • I have tried to change both net.inet.tcp.msl system parameter and MaxIdleConnsPerHost (this way`proxy.Transport = &mimTransport{MaxIdleConnsPerHost: 10}`) it seems have no effect on `connection reset by peer` error. I also switched to vegeta, and it gives me much better results. But error still occurs when i set keepalive to false (real http requests usually do not use keepalive). – Mark Belotskiy Jan 11 '17 at 14:43

0 Answers0