Apache HttpAsyncClient memory leak

Question

I need to use HttpAsyncClient under the high load. I create HttpAsyncClient like this:

RequestConfig requestConfig = RequestConfig.custom()
        .setConnectTimeout(CONNECT_TIMEOUT)
        .setSocketTimeout(SOCKET_TIMEOUT)
        .setConnectionRequestTimeout(CONNECTION_REQUEST_TIMEOUT)
        .build();
HttpAsyncClient client = HttpAsyncClients.custom()
        .setDefaultRequestConfig(createRequestConfig())
        .build();

And then I use it like this:

HttpPost request = new HttpPost(url);
request.setEntity(new StringEntity(requestBody, "UTF-8"));
client.execute(request, null)

Usually I don't really care about response, so I don't initialize Future<HttpResponse> variable and don't do Future.get(). Well, just for the clarification (I don't think it has something to do with question), I care about responses sometimes, but 99% of responses are not interesting for me.

The problem is when I do a lot of requests (for instance 300 every second, and by "request" word here I mean client.execute() invokation) I finally get java.lang.OutOfMemoryError: GC overhead limit exceeded. I tried to use VisualVM to find out what's happening. I see that java.lang.Object[], char[], java.lang.String, byte[], short, char[] instances count is growing (I tried to force GC and to limit heap size to be sure it isn't normal - didn't help). And so is growing the used heap space.

What causes this problem? Maybe I should use HttpAsyncClient some different way? Do I need to use custom RequestProducer, ResponseProducer or to use CountDownLatch?

UPD The problem was because of PowerMock library

Please see this link http://stackoverflow.com/questions/4775618/httpclient-4-0-1-how-to-release-connection You will find resolution in this topic. — eg04lt3r, Jan 15 '16 at 14:53
Connection management in the classic and async HCs are completely different. Everything what has been said in 4775618 bears no relevance for HttpAsyncClient — ok2c, Jan 15 '16 at 15:25

ok2c · Answer 1 · 2016-01-15T14:52:38.340

4

HttpAsyncClient does not throttle request execution rate in any way or fashion to avoid blocking of the #execute method. One can submit boundless number of requests to the client and the client will dutifully stick them all into the execution queue. How fast those requests can actually be executed and removed from the queue is a whole different story and can depend on many factors. In your particular case you are trying to process all those requests with just 2 concurrent connections per route, which may not necessarily the optimal thing to do.

PS: /old man's grumbling/ people should not presume that async clients would be somehow faster for some reason. They will not be unless used for specific use cases and in a specific manner. What is certain that one can easily end up using more memory with async clients unless very careful.

edited Jan 15 '16 at 14:52

answered Jan 15 '16 at 14:47

ok2c

26,450
5
63
71

"does not throttle request execution rate in any way or fashion to avoid blocking of the #execute method" - cound you please explain it a little more? I was thinking about these requests queue inside client too. But here is the thing - I set `connectionRequestTimeout` so I thought I can be sure that if request in queue don't get it's own connection in 5 seconds, it is cancelled with `TimeoutException`. But somehow it looks like GC can't clean something inside client. – coolguy Jan 15 '16 at 15:32
Some offtop about performance: in my case first advantage of using async client is that thread won't be blocked; and second, maybe, pipelining support. Ofc I could create some kind of facade over the synchronous client, but it would be much harder - cancellation support, some other specific features etc – coolguy Jan 15 '16 at 15:36
Instead of guessing you should find out what exactly does not get cleaned up. – ok2c Jan 15 '16 at 16:01
Any ideas how could I check it? I've took a look at `HttpAsyncClient`'s source code and can't really find out what could not be erased. As far as I understand `InternalHttpAsyncClient` (it's the implementation I use at the moment) creates a new `DefaultClientExchangeHandlerImpl` and starts it. It tries to get the connection from the pool until the timeout is reached. And there I can't see any problems that could lead to obsolete references or something. Anyway seems like there aren't any API I can control this outside with. Am I using `HttpAsyncClient` right in my question? – coolguy Jan 17 '16 at 07:27
I've wrote a simple load test: http://pastebin.com/0XaquW35 . Here is the part of output: http://pastebin.com/ykCyJNHY . It was unexpected for me that none of the requests have been executed. If I sleep 1000ms every iteration in my test, it works perfectly fine. It's a strange way that `HttpAsyncClient` manage requests. Considering that old requests are being removed from execution queue after `connectionRequest` timeout I can't understand is there any chance that queue size is growing constantly. Btw, by `execution queue` phrase you meant many instances of `DefaultClientExchangeHandlerImpl`? – coolguy Jan 17 '16 at 07:35
It's just for instance, I needed responsive host for this sample test. Could you take a look at another questions please? – coolguy Jan 17 '16 at 10:48
Why? What is the point in me telling you that one should be using a profiler or memory analyzer when investigating a potential memory leak? – ok2c Jan 17 '16 at 13:12
Ok, I'll try to profile it again. Well as far as I understand my problem is about `HttpAsyncClient` cannot manage too many requests and "chokes". I just found your answer here https://stackoverflow.com/questions/30101865/how-to-configure-the-number-of-allowed-pending-requests-in-apache-async-client think it's what I need, I'll try tomorrow – coolguy Jan 17 '16 at 14:52
HttpAsyncClient does exactly what your code instructs it to do. Your problem is that you do not seem to have a clear idea what you are trying to do – ok2c Jan 17 '16 at 16:12
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/100935/discussion-between-coolguy-and-oleg). – coolguy Jan 18 '16 at 06:33

score 0 · Accepted Answer · answered Jan 21 '16 at 11:04

0

The reason was that I were using PowerMock to run my tests. Without PowerMock everything works fine.

answered Jan 21 '16 at 11:04

coolguy

3,011
2
18
35

Apache HttpAsyncClient memory leak

2 Answers2