Why is Go HTTPS Client not reusing connections?

Question

I have an http client which creates multiple connections to the host. I want to set a maximum number of connections it can set to a particular host. There are no such options in go's request.Transport. My code looks like

package main 

import (
  "fmt"
  "net/http"
  "net/url"
)

const (
  endpoint_url_fmt      = "https://blah.com/api1?%s"
)


func main() {

  transport := http.Transport{ DisableKeepAlives : false }

  outParams := url.Values{}
  outParams.Set("method", "write")
  outParams.Set("message", "BLAH")

  for {
    // Encode as part of URI.
    outboundRequest, err := http.NewRequest(
      "GET",
      fmt.Sprintf(endpoint_url_fmt, outParams.Encode()),
      nil
    )
    outboundRequest.Close = false
    _ , err = transport.RoundTrip(outboundRequest)
    if err != nil {
      fmt.Println(err)
    }
  }

}

I would expect this to create 1 connection. As I am calling it in a for-loop. But this keeps creating an infinite number of connections.

Where as similar python code using the requests library creates only one connection.

#!/usr/bin/env python
import requests
endpoint_url_fmt      = "https://something.com/restserver.php"
params = {}
params['method'] = 'write'
params['category'] = category_errors_scuba
params['message'] = "blah"
while True:
  r = requests.get(endpoint_url_fmt, params = params)

For some reason the go code is not reusing http connections.

EDIT : The go code needs the body to be closed to reuse the connection.

 resp , err = transport.RoundTrip(outboundRequest)
 resp.Close() //  This allows the connection to be reused

I do not think `MaxIdleConnsPerHost` is doing what you think it's doing. It's meant to support HTTP 1.1 Keep-Alives, not limit the number of hosts you connect to. It also seems like you're doing the `HTTP GET` in a slightly raw way. Can you adapt http://golang.org/pkg/net/http/#example_Get instead? — dyoo, Jul 31 '13 at 04:27
Edited question simplified example with only a for loop. No go routines. Added equivalent python code, which does the right thing. — sheki, Aug 01 '13 at 05:00
Relevant discussions: https://github.com/google/go-github/pull/317 — Arnie97, Apr 22 '21 at 07:29

Matt Self · Accepted Answer · 2014-03-01T18:42:01.920

Based on further clarification from the OP. The default client does reuse connections.

Be sure to close the response.

Callers should close resp.Body when done reading from it. If resp.Body is not closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.

Additionally, I've found that I also needed to read until the response was complete before calling Close().

e.g.

res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body)
res.Body.Close()

To ensure http.Client connection reuse be sure to do two things:

Read until Response is complete (i.e. ioutil.ReadAll(resp.Body))
Call Body.Close()

Old answer, useful for rate limiting, but not what the OP was after:

I don't think setting max connections is possible via the golang 1.1 http APIs. This means you can shoot yourself in the foot with tons of TCP connections (until you run out of file descriptors or whatever) if you aren't careful.

That said, you could limit the rate at which you call the go routine for a particular host (and therefore outbound requests and connections) via time.Tick.

For example:

import "time"

requests_per_second := 5
throttle := time.Tick(1000000000 / requests_per_second)

for i := 0; i < 16; i += 1 {
  <-throttle  
  go serveQueue()
}

I added a python equivalent code. that piece of code does what is expected, open only one connection. — sheki, Aug 01 '13 at 05:01
The way the question was written (i.e. "set maximum number") led me to believe you were attempting to limit outbound connections not re-use them. I can now infer that you want to reuse connections to the degree that is possible. You just need to set the Keep-Alive header The python requests library handles keep-alive for you, while go does not. I'll add a snippet to my answer. — Matt Self, Aug 01 '13 at 15:32
Go client uses http 1.1 by default. Setting the header is not the issue. Go needs the Response Body to be closed for a connection to be reused. — sheki, Aug 01 '13 at 17:40
Yep. I didn't spot it. Glad it works now. Updated my answer for future SO users... hopefully it will help someone. — Matt Self, Aug 01 '13 at 18:28

score 1 · Answer 2 · answered Jan 22 '20 at 08:22

There are interesting improvements in http.Transport:

// DisableKeepAlives, if true, disables HTTP keep-alives and
// will only use the connection to the server for a single
// HTTP request.
//
// This is unrelated to the similarly named TCP keep-alives.
DisableKeepAlives bool

// ...

// MaxIdleConns controls the maximum number of idle (keep-alive)
// connections across all hosts. Zero means no limit.
MaxIdleConns int // Go 1.7

// MaxIdleConnsPerHost, if non-zero, controls the maximum idle
// (keep-alive) connections to keep per-host. If zero,
// DefaultMaxIdleConnsPerHost is used.
MaxIdleConnsPerHost int

// MaxConnsPerHost optionally limits the total number of
// connections per host, including connections in the dialing,
// active, and idle states. On limit violation, dials will block.
//
// Zero means no limit.
MaxConnsPerHost int // Go 1.11

Why is Go HTTPS Client not reusing connections?

2 Answers2

Linked