Webclient Leaks Memory In Docker?

Question

I'm trying to use Webclient in my project, but when I load test, I'm noticing the docker memory usage never goes down until the instance dies.

@Component
public class Controller {

  //This is an endpoint to another simple api
  //I use my local Ip instead of localhost in the container
  private static final String ENDPOINT = "http://localhost:9090/";

  private WebClient client;
  
  public Controller(WebClient.Builder client) {
    super();
    this.client = client.build();
  }

  @Bean
  public RouterFunction<ServerResponse> router() {

    return RouterFunctions.route(GET("helloworld"), this::handle);
  }

  Mono<ServerResponse> handle(ServerRequest request) {

    Mono<String> helloMono =
        client.get().uri(ENDPOINT + "/hello").retrieve().bodyToMono(String.class);

    Mono<String> worldMono =
        client.get().uri(ENDPOINT + "/world").retrieve().bodyToMono(String.class);

    return Mono.zip(helloMono, worldMono, (h, w) -> h + w)
        .flatMap(s -> ServerResponse.ok().bodyValue(s));
  }
}

Here's my dockerFile as well.

FROM openjdk:8

ENV SERVICE_NAME reactive-hello-world

ADD target/reactive-hello-world-*.jar $APP_HOME/reactive-hello-world.jar

RUN mkdir /opt/reactor-netty/


EXPOSE 9010

CMD java \
    -Dcom.sun.management.jmxremote=true \
    -Dcom.sun.management.jmxremote.local.only=false \
    -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.ssl=false \
    -Djava.rmi.server.hostname=localhost \
    -Dcom.sun.management.jmxremote.port=9010 \
    -Dcom.sun.management.jmxremote.rmi.port=9010 \ 
    -Xmx190M \
    -jar reactive-hello-world.jar

EXPOSE 8080

Have I missed a step somewhere?

Edit: Here's some images

Before Load Test:

After Load Test

As you can see, the GC is happening correctly but the memory hasn't decreased. If I let the test continue it kills the instance in a couple minutes.

I've tried similar code using RestTemplate and I'm not experiencing any issues, the memory doesn’t usually exceed 400MB even when I run the Jmeter for an extended time. Can you help understand what's happening?

Edit: I've also tried the deprecated AsyncRestTemplate and I'm not seeing a problem with that either.

Edit: I have created the repos for this example. Please check if you can reproduce the issue.

The Hello World Backend

The Webclient Hello World(JMX is inside this repo)

The RestTemplate Hello World

The AsyncRestTemplate Hello World

so you are calling the endpoint `/hello` that in turn calls endpoint `/hello` in an endless loop? and then it crashes... or am i missing something? — Toerktumlare, Aug 17 '20 at 23:35
I created a second API on port 9090 that has 2 simple string endpoints. — Jojo, Aug 18 '20 at 05:41
"memory usage never goes down" - does it ever reach the point of OutOfMemoryError? If not then it is possible that garbage collection not kicked in yet. Try to run your test until you reach OOM or the used memory goes down. Also, if you're looking into performance: you shouldn't create a new WebClient for each request, you should autowire a single WebClient instance or create it as static field in the class. — Martin Tarjányi, Aug 18 '20 at 14:42
It doesn't go down, I've checked the GC was happening with visualVM and still the memory was not dropping. The heap memory went down but the docker memory was barely affected. — Jojo, Aug 18 '20 at 16:22
What makes you think that there’s a problem, and that it’s caused by `WebClient`? — Abhijit Sarkar, Aug 21 '20 at 02:24
Problem is that our containers are running out of memory, and through a systematic bug hunt, we found that the memory increase is directly correlated to ``WebClient`` calls. I was able to reproduce the issue with the simple projects above. As I've said before, when I replaced the ``WebClient`` calls with the blocking ``RestTemplate``, we don't face any issues with the memory, which naturally makes me think ``Webclient`` is the issue.. — Jojo, Aug 21 '20 at 02:52

Jojo · Accepted Answer · 2020-10-08T15:43:43.490

1

Nevermind lads, I was able to find the answer, see:

https://github.com/reactor/reactor-netty/issues/1304

Essentially, the reactor netty dependency was outdated.

edited Oct 08 '20 at 15:43

answered Sep 29 '20 at 20:44

Jojo

76
1
6

1

You may want to copy some of the details over to your answer – Darren Forsythe Sep 29 '20 at 23:09

score 0 · Answer 2 · answered Aug 21 '20 at 00:41

I don't think your problem has anything to do with the RestTemplate vs WebClient. Your GC graph looks pretty normal. It doesn't look like there is a leak, because whenever a GC happens, it seems like the allocated memory is able to go back to the previous levels.

It is important to note that you wouldn't always see a drop in the container memory usage when a garbage collection happens. This is because the Java Virtual Machine does not necessarily return memory back to system after a GC. This means, even if a portion of the memory is unused/freed after a GC in your process, it may still appear as "used" from outside of the process.

To be clear: JVM does return memory back to the system in some circumstances, and it depends on various factors including the garbage collector that is used.

From the GC graph on the second screenshot of your example, the GC graph looks pretty normal, but it seems that a pretty small amount of heap was released back to the system (the orange area).

You can try switching to G1 GC via -XX:+UseG1GC JVM flag and tune the behavior to more aggressively release the unallocated memory back to the system by tweaking -XX:MinHeapFreeRatio and -XX:MaxHeapFreeRatio, particularly by reducing the -XX:MaxHeapFreeRatio value from the default 70 percent to a lower number. See here.

Lowering -XX:MaxHeapFreeRatio to as low as 10% and -XX:MinHeapFreeRatio has shown to successfully reduce the heap size without too much performance degradation; however, results may vary greatly depending on your application. Try different values for these parameters until they're as low as possible, yet still retain acceptable performance.

For further information on the topic, see:

Does GC release back memory to OS?

JEP 346: Promptly Return Unused Committed Memory from G1

If you notice in the docker file I reduced the maximum heapSize to 190MB. And you see in the graph that the heap never exceeds this value. How then does the memory usage never go down if not for some offheap leak? If I run the jmx long enough it eventually kills the docker instance. Also, if webclient isn't the issue, why doesn't this happen when I use restTemplate? — Jojo, Aug 21 '20 at 01:45
@Jojo I see. It is possible that on the project with webclient there are more classes loaded by the classloader due to reactive streams. You can verify this by looking at the number of loaded classes in your screenshots. So the difference might be caused by the metaspace size. Can you check that? — Utku Özdemir, Aug 21 '20 at 08:57
The metaspace size looks normal as well. My most recent run had the instance memory go up to 1GB. looking at the VisualVM, I can't see any obvious cause. — Jojo, Aug 21 '20 at 14:06

Webclient Leaks Memory In Docker?

2 Answers2