8

I am using Spring WebClient to call a rest service. The code for the post call as mentioned below.

Mono<ClientResponse> response = client.post()
                        .uri(uriBuilder -> uriBuilder.build())
                        .headers(httpHeaders -> httpHeaders.setAll(getHeaders()))
                        .body(BodyInserters.fromPublisher(Mono.just(message), String.class))
                        .exchange();
                response.subscribe(clientResponse -> {
                   System.out.println(clientResponse.statusCode());
                });

I am getting the following exception after posting continuously for a while (after posting 2-3 million requests in 5 mins). 

 [     parallel-3] r.c.s.Schedulers                         : Scheduler worker in group main failed with an uncaught exception

reactor.core.Exceptions$ErrorCallbackNotImplemented: reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
Caused by: reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
    at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:317) ~[reactor-netty-0.9.0.RELEASE.jar!/:0.9.0.RELEASE]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
    |_ checkpoint ⇢ Request to POST https://myrest-service-c12-1-lb-125370128.us-west-2.elb.amazonaws.com/antenna [DefaultWebClient]
Stack trace:
        at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:317) ~[reactor-netty-0.9.0.RELEASE.jar!/:0.9.0.RELEASE]
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) [reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) [reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]

It looks like the pool is exhausted and I need to limit the number of requests. Could some one help me in solving this issue. Thanks in advance.

mnpr
  • 325
  • 4
  • 8
  • Why do you think it's because the pool is exhausted? From the exception message, the error can be caused by the server side as well. have you checked server side log? – danny Dec 22 '19 at 11:45
  • There is no error in the server logs. I think because the client is pushing millions of messages, the Webclient is running out of connections. – mnpr Dec 22 '19 at 18:03
  • This error stopped when I throttle the number of requests from the client end. – mnpr Dec 26 '19 at 21:45
  • what kind of `ConnectionProvider` are you using when create your http client? at first I used a fixed connection provider and have the same issue with you. Now I change to an elastic provider which can get more connections dynamically. After changing, I don't have this `PoolAcquireTimeoutException` so maybe you can try that as well – danny Dec 27 '19 at 03:25
  • 1
    I was simply using.. WebClient.builder().clientConnector(new ReactorClientHttpConnector(getHttpClient())).baseUrl(url).build(); I updated it to... ConnectionProvider elasticPool = ConnectionProvider.elastic("ElasticPool"); HttpClient httpClient = HttpClient.create(fixedPool); WebClient.builder().clientConnector(new ReactorClientHttpConnector(httpClient)).baseUrl(url).build(); It simply hanged after posting 5000+ messages. – mnpr Dec 27 '19 at 21:07
  • 1
    When I used the fixed pool with higher number of connections, I experienced the same above issue... ConnectionProvider fixedPool = ConnectionProvider.fixed("fixedPool", 10000); After I changed the number of connections from 10000 to 1000, it started posting, but had the actual issue (PoolAcquireTimeoutException) came again. After experimented with different number of connections, I got settled with 1000 and request throttling. – mnpr Dec 27 '19 at 21:08

1 Answers1

1

Seems to be a bug in reactor-netty that has recently been fixed: https://github.com/reactor/reactor-netty/issues/1012

Thilo-Alexander Ginkel
  • 6,898
  • 10
  • 45
  • 58