Rest Template - No buffer space available (maximum connections reached?)

Question

I'm using RestTemplate to execute concurrent HTTP requests. After some time, I start getting

java.net.SocketException: No buffer space available (maximum connections reached?)

I know that this is related to sockets in TIME_WAIT status.
I have tried installing Windows 7 fix, that most sources encourage.

I configured RestTemplate to use HttpClient as follows:

val httpClient = HttpClientBuilder.create()
            .setMaxConnPerRoute(properties.concurrencyLimit)
            .setMaxConnTotal(properties.concurrencyLimit)
            .build()
return RestTemplate(HttpComponentsClientHttpRequestFactory(httpClient))

I have tried to use billion other HttpClient configurations
I have tried different amounts of concurrent requests between 20-100

Just before I was going to press Post your question button, my colleague found a solution, that doesn't make any sense to me:

 val httpClient = HttpClientBuilder.create()
            .setMaxConnPerRoute(properties.concurrencyLimit * 2)
            .setMaxConnTotal(properties.concurrencyLimit * 2)
            .build()
    return RestTemplate(HttpComponentsClientHttpRequestFactory(httpClient))

Basically, when I set connection pool twice as big as threads number, the whole thing works like a charm.

WHY? WHY DOESN'T FIRST CONFIGURATION WORK AND SECOND DOES?

All dependencies are managed by Spring Boot 1.4.0.RELEASE parent pom.

If you use Async connector or async servlet then while you are making you rest call the thread can be serving other requests, meaning when nex request comes and tries to get connection to make http call there will be no connections left — Dennis R, Sep 15 '16 at 17:54
It's a batch job. All it does is: it hits two endpoints and combines the results, prints a report at the end. — grzegorztj, Sep 16 '16 at 07:24
In your batch do you do http requests one by one? Then in http 1.1 if the current http(under underneath tcp) connection is busy doing http request other reuest will take new connection, there is no multiplexing till http 2.0. So your batch will use multiple http connections and they can be exhausted faster then your threads processing batches. — Dennis R, Sep 16 '16 at 13:24
That's why I wanted to use a connection pool. My current best theory is that when hitting 2 services interchangeably and connections to each are not distributed 50/50, then some connections are going to be "discarded". When enough connections have been discarded and their sockets are in TIME_WAIT, the pool cannot establish any new ones. What do you think? — grzegorztj, Sep 19 '16 at 07:51

score 1 · Accepted Answer · edited May 23 '17 at 12:00

Connections are pooled based on a route. Since you have at least 2 routes, total number of connections should be at least 2 times of max connections per route. See also other related post And yes, I would suggest to make even conenctionPool per rote more then number of threads if you process your batch one by one or there is some async nature.(not exactly sure how you use concurrencyLimit)

See except from here:

PoolingHttpClientConnectionManager is a more complex implementation that manages a pool of client connections and is able to service connection requests from multiple execution threads. Connections are pooled on a per route basis. A request for a route for which the manager already has a persistent connection available in the pool will be serviced by leasing a connection from the pool rather than creating a brand new connection. PoolingHttpClientConnectionManager maintains a maximum limit of connections on a per route basis and in total. Per default this implementation will create no more than 2 concurrent connections per given route and no more 20 connections in total. For many real-world applications these limits may prove too constraining, especially if they use HTTP as a transport protocol for their services.

Rest Template - No buffer space available (maximum connections reached?)

1 Answers1