4

I got this error all of a sudden in production while the application was not under any load.

The issue happened when my code tries to send the PUT message using spring rest template

Here is the code how I am initialing the restTemplate

private static final RestTemplate restTemplate = new RestTemplate(new HttpComponentsClientHttpRequestFactory());
{

    List<HttpMessageConverter<?>> messageConverters = new ArrayList<HttpMessageConverter<?>>();
    Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
    marshaller.setClassesToBeBound(PaymentSession.class);
    MarshallingHttpMessageConverter marshallingHttpMessageConverter = new MarshallingHttpMessageConverter(marshaller, marshaller);
    marshallingHttpMessageConverter.setSupportedMediaTypes(Arrays.asList(MediaType.APPLICATION_XML, MediaType.TEXT_HTML));
    messageConverters.add(marshallingHttpMessageConverter);
    restTemplate.setMessageConverters(messageConverters);
}

Call to PUT

try {
    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(MediaType.APPLICATION_XML);
    HttpEntity<PaymentSession> httpEntity = new HttpEntity<PaymentSession>(session, headers);

    restTemplate.exchange(baseUrl+"/v1/psps", HttpMethod.PUT, httpEntity, PaymentSession.class);

}catch(HttpClientErrorException e){
        logger.error("Exception..!!",e)
}

Exception stacktrace

Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:232)
at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:199)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:456)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:88)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:46)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:49)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:509)
NullPointerException
  • 3,732
  • 5
  • 28
  • 62

3 Answers3

12

I would suggest to configure HttpComponentsClientHttpRequestFactory instance being passed in the constructor of RestTemplate increasing defaultMaxPerRoute or maxPerRoute for the specific http route for which requests are timing out, increasing the pool size is not enough, as I mentioned in a comment, even if you set PoolingHttpClientConnectionManager.setMaxTotal() to 200, HttpComponentsClientHttpRequestFactory uses a defaultMaxPerRoute of 4, my guess would be in an attempt for a host route (scheme, host, port) not to hijack the connection pool)

...
public PoolingHttpClientConnectionManager poolingHttpClientConnectionManager() {
        PoolingHttpClientConnectionManager result = new PoolingHttpClientConnectionManager();
        result.setMaxTotal(this.httpHostConfiguration.getMaxTotal());
        // Default max per route is used in case it's not set for a specific route
        result.setDefaultMaxPerRoute(this.httpHostConfiguration.getDefaultMaxPerRoute());
        // and / or
        if (CollectionUtils.isNotEmpty(this.httpHostConfiguration.getMaxPerRoutes())) {
            for (HttpHostConfiguration httpHostConfig : this.httpHostConfiguration.getMaxPerRoutes()) {
                HttpHost host = new HttpHost(httpHostConfig.getHost(), httpHostConfig.getPort(), httpHostConfig.getScheme());
                // Max per route for a specific host route
                result.setMaxPerRoute(new HttpRoute(host), httpHostConfig.getMaxPerRoute());
            }
        }
        return result;
    }
  ...


@Configuration
@ConfigurationProperties(prefix = "httpConnPool")
public class HttpHostsConfiguration {

  private Integer maxTotal;
  private Integer defaultMaxPerRoute;
  private List<HttpHostConfiguration> maxPerRoutes;

  // Getters, Setters
...

application.yml

httpConnPool:
  maxTotal: 20
  defaultMaxPerRoute: 20
  maxPerRoutes:
    -
      scheme: http
      host: localhost
      port: 8800
      maxPerRoute: 20

I recently blog about Troubleshooting Spring's RestTemplate Requests Timeout where requests timing out were troubleshooted using JMeter and shell commands and fixed via configuration settings.

ootero
  • 3,235
  • 2
  • 16
  • 22
10

Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool

This error is self describing. You need to increase your connection pool in production - current implementation of HttpComponentsClientHttpRequestFactory default constructor is using HttpClientBuilder with .useSystemProperties().

I believe it will be 5 connections by default. This works for client but is unlikely what you want in server environment. You need to use something like

new RestTemplate(new HttpComponentsClientHttpRequestFactory(HttpClientBuilder.create()
                    .setMaxConnTotal(200)
                    .setMaxConnPerRoute(50)
                    .build()));
Boris Treukhov
  • 17,493
  • 9
  • 70
  • 91
  • I suspect that this is the issue with connection pool size as I mentioned that the issue occurred under very minimal load and we didn't faced this when we did the load test for 50 concurrent users. But yes we are going to increase the pool size from 5 to 25 (4 servers i.e. 100 connections) – NullPointerException Jul 24 '15 at 13:03
  • @NullPointerException please set the timeout values also - it may be that in case of failure connection takes several seconds so it goes out of the pool. – Boris Treukhov Jul 24 '15 at 14:51
  • @NullPointerException 30 seconds is way too high - 2-3 seconds is what I can imagine - if the servers are in the different parts of the Earth. – Boris Treukhov Jul 24 '15 at 18:11
  • 6
    One thing to take into account is that it doesn't matter how big is the pool if you don't set .setMaxConnPerRoute(), which I think is by default 4, it will only use 4 connections for the same host, even though you might have another 196 connections idle, I guess to prevent a host from hijacking the pool. – ootero Dec 23 '16 at 15:58
  • 1
    @BorisTreukhov No it is not too high. There is ConnectTimeout (how much time I need for connection establishment) and ReadTimeout (how much should I wait for the request completion). If I read 10MB on the dialup link (just example), then 30secs for ReadTimeout is maybe too low. – Mejmo Mar 29 '17 at 18:39
  • @Mejmo it's 2017, and our task is to produce decent services, by no way front end service which services results longer than 1 second is satisfactory, and the requirements of the back-end services are even stronger(sometimes it's nanoseconds who are in count not even milliseconds) - if you are expecting long result then your service should be asynchronous(and you should consider using reactive programming in this case) Also this question is about back-end processing, it may be OK for last mile GPRS connection but it's unlikely that your server will be using GPRS to connect to another server. – Boris Treukhov Mar 29 '17 at 21:19
1

The problem is the HTTP client connections aren't being closed. I had the same problem with a service that only has about 1 request per second. 

"exception":"org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool"

You need to add a finally block and close the connection.

cwa
  • 166
  • 2
  • 11