Turning off connection pool for Go http.Client

Question

For testing purposes I'm trying to create a net/http.Client in Go that has connection pooling disabled. What I'm trying to achieve is a new TCP connection is established to the address on every HTTP/1.x request.

Currently I have:

        c = &http.Client{
            Transport: &http.Transport{
                DialContext: (&net.Dialer{
                    Timeout:   5 * time.Second,
                    KeepAlive: 5 * time.Second,
                }).DialContext,
                TLSHandshakeTimeout:   5 * time.Second,
                ResponseHeaderTimeout: 5 * time.Second,
                ExpectContinueTimeout: 1 * time.Second,
            },
        }

Any ideas how should I tweak this?

I'm seeing that if I set c.Transport.MaxIdleConns = 1 this could work, but I'm not exactly sure if this still allows 1 in-use + 1 idle (2 total) TCP connections:

    // MaxIdleConns controls the maximum number of idle (keep-alive)
    // connections across all hosts. Zero means no limit.
    MaxIdleConns int

Similarly, it seems like c.Dialer.KeepAlive = -1 could do this, too:

    // KeepAlive specifies the keep-alive period for an active
    // network connection.
    // If zero, keep-alives are enabled if supported by the protocol
    // and operating system. Network protocols or operating systems
    // that do not support keep-alives ignore this field.
    // If negative, keep-alives are disabled.

but I'm not sure about the behavior for TCP connections + Keep-Alive + HTTP.

Another approach is to try to kill idle TCP connections as soon as possible, so I set c.Transport.IdleConnTimeout = 1*time.Nanosecond.

When I did this, my Client.Do() now occassionally returns error:

tls: use of closed connection

I'm suspecting this is a Go stdlib issue (perhaps a race) that it uses a connection that should've been taken out of a pool.

Answer 1

Connections are added to the pool in the function Transport.tryPutIdleConn. The connection is not pooled if Transport.DisableKeepAlives is true or Transport.MaxIdleConnsPerHost is less than zero.

Setting either value disables pooling. The transport adds the Connection: close request header when DisableKeepAlives is true. This may or may not be desirable depending on what you are testing.

Here's how to set DisableKeepAlives:

t := http.DefaultTransport.(*http.Transport).Clone()
t.DisableKeepAlives = true
c := &http.Client{Transport: t}

Run a demonstration of DisableKeepAlives = true on the playground.

Here's how to set MaxIdleConnsPerHost:

t := http.DefaultTransport.(*http.Transport).Clone()
t.MaxIdleConnsPerHost = -1
c := &http.Client{Transport: t}

Run a demonstration of MaxIdleConnsPerHost = -1 on the playground.

The code above clones the default transport to ensure that the default transport options are used. If you explicitly want the options in the question, then use

    c = &http.Client{
        Transport: &http.Transport{
            DialContext: (&net.Dialer{
                Timeout:   5 * time.Second,
                KeepAlive: 5 * time.Second,
            }).DialContext,
            TLSHandshakeTimeout:   5 * time.Second,
            ResponseHeaderTimeout: 5 * time.Second,
            ExpectContinueTimeout: 1 * time.Second,
            DisableKeepAlives: true,
        },
    }

or

    c = &http.Client{
        Transport: &http.Transport{
            DialContext: (&net.Dialer{
                Timeout:   5 * time.Second,
                KeepAlive: 5 * time.Second,
            }).DialContext,
            TLSHandshakeTimeout:   5 * time.Second,
            ResponseHeaderTimeout: 5 * time.Second,
            ExpectContinueTimeout: 1 * time.Second,
            MaxIdleConnsPerHost: -1,
        },
    }

MaxIdleConnsPerHost does not limit the number of active connections per host. See this playground example for a demonstration.

Pooling is not disabled by setting Dialer.KeepAlive to -1. See this answer for an explanation.

Answer 2

You need to set the DisableKeepAlives to true, and MaxIdleConnsPerHost to -1.

From the documentation:

// DisableKeepAlives, if true, disables HTTP keep-alives and
// will only use the connection to the server for a single
// HTTP request.

https://golang.org/src/net/http/transport.go, line 166 and 187

So, your client have to be initialized as following

c = &http.Client{
    Transport: &http.Transport{
        DialContext: (&net.Dialer{
            Timeout:   5 * time.Second,
        }).DialContext,
        TLSHandshakeTimeout:   5 * time.Second,
        ResponseHeaderTimeout: 5 * time.Second,
        ExpectContinueTimeout: 1 * time.Second,
        DisableKeepAlives: true,
        MaxIdleConnsPerHost: -1
    },
}

If you are using a Go version prior than 1.7, than you need to consume all the buffer of the body and only after call the request.Body.Close(). Instead, if you are using a version greater or equal the 1.7, you can defer the close without additional precautions.

Example library that has connection pooling disabled, but is still able to perform parallel requests: https://github.com/alessiosavi/Requests

Answer 3

The http.Transport has a property called MaxConnsPerHost

MaxConnsPerHost optionally limits the total number of connections per host, including connections in the dialing, active, and idle states.

It includes the dialing, active and idle states