Micronaut Read Timeout with HttpClient

Question

I'm struggling to use the Micronaut HTTPClient for multiple calls to a third-party REST service without receiving a io.micronaut.http.client.exceptions.ReadTimeoutException

To remove the third-party dependency, the problem can be reproduced using a simple Micronaut app calling it's own service.

Example Controller:

@Controller("/")
public class TestController {
      
    @Inject
    private TestClient client;

    @Get("service")
    String service() {
        return "Hello World Service";
    }
    @Get("mproxy")
    String multiproxy() {
        StringBuffer sb = new StringBuffer();
        for(int i=0;i<20;i++){
            sb.append(client.getService());
        }
        return sb.toString();
    }
    @Get("proxy")
    String proxy() {
        return client.getService();
    }  
}

Test Client:

@Client("http://localhost:8080")
public interface TestClient {
    
    @Get("/service")
    String getService();

}

Calling the /service end-point directly from using curl, ab or postman produces no errors.

Calling the /mproxy end-point will throw an exception

ERROR i.m.r.intercept.RecoveryInterceptor - Type [clienttest.TestClient$Intercepted] executed with error: Read Timeout
io.micronaut.http.client.exceptions.ReadTimeoutException: Read Timeout
        at io.micronaut.http.client.exceptions.ReadTimeoutException.<clinit>(ReadTimeoutException.java:26)
        at io.micronaut.http.client.netty.DefaultHttpClient$12.exceptionCaught(DefaultHttpClient.java:2316)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424)
        at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92)
        at io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:145)
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
        at io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:231)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
        at io.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:98)
        at io.netty.handler.timeout.ReadTimeoutHandler.channelIdle(ReadTimeoutHandler.java:90)
        at io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504)
        at io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476)
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:831)

Alternatively, the same exception is thrown if the /proxy endpoint is tested via ab

ab -c 5 -n 200 localhost:8080/proxy

or via mulitple calls with postman.

This is for micronaut version 2.5.5 with an absolutely vanilla template app, no connection-pooling or time-outs specified in application.yml.

It appears to error after 4 connections/clients but changing connection-pooling and timeouts seems to not change the result. Am I missing some client config?

Answer 1

If this isn't going to throw an exception then I don't know what is going to.

This is caused by using blocking code within Netty's event loop.

The code over here is making a blocking request 20 times in a row which cause the machine to break. I don't know what data is coming from the client but I would never recommend to do it in this manner.

 for(int i=0;i<20;i++){
        sb.append(client.getService());
    }

Key message: don't block the event loop

To solve this what you can do is make your request Asynchronous. To do this make use of RxJava. RxJava allows you to perform the operations in asynchronous manner. It provides you some very useful observables and operators.

The only other way: Run this operator on another thread so, that the main thread doesn't get blocked but this may not work very efficiently and still cause problem.

To get start with RxJava follow the link: https://factoryhr.medium.com/understanding-java-rxjava-for-beginners-5eacb8de12ca

Micronaut Tutorial Reactive: https://piotrminkowski.com/2019/11/12/micronaut-tutorial-reactive/

Answer 2

Update to accepted answer above, just to provide examples of working code. There are two options to not block the event-loop - use Reactive return types or execute the proxy endpoint on a different thread loop - example:

@Controller("/")
public class TestController {
   
    
    @Inject
    private TestClient client;

    @Inject
    private RXTestClient rxclient;

    @Get("rxservice")
    Single<String> rxservice() {
        return Single.just("Hello World Service");
    }
    @Get("service")
    String service() {
        return "Hello World Service";
    }

   
    @Get("rxproxy")
    Single<String> rxproxy() {
        return rxclient.getService();
    }
   
    @ExecuteOn(TaskExecutors.IO)
    @Get("proxy")
    String proxy() {
        return client.getService();
    }
}