8

I have a third party API, which I call using an HTTP GET request. Each request takes a few seconds to get a response.

Currently I am using a CompletableFuture which I am executing on a FixedThreadPool of size 64. This is causing the threads to be blocked until it recieves a response for the GET request, i.e. the threads sit idle after sending the GET response until they recieve a response. So my maximum number of simultaneous requests I can send out is limited by my thread size i.e. 64 here.

What can I use instead of CompletableFuture so that my threads don't sit idle waiting for the response?

Aayush Dwivedi
  • 83
  • 1
  • 5
  • So increase the thread pool, or use an unlimited one. But do you really have 64 pending GET requests pending at the same time? – user207421 Mar 22 '20 at 06:23
  • 1
    I can actually have several thousands of GET requests simultaneously. Is there another way instead of using a thread pool of such size? The threads just have to send the GET request and collect the response. – Aayush Dwivedi Mar 22 '20 at 06:28
  • There's nothing wrong with several thousand idle threads, and non-blocking HTTP is seriously difficult to implement. – user207421 Mar 22 '20 at 06:36
  • The issue we have with having multiple threads is that we might need to keep the size of thread pool changing as we scale up. And also the thread size we can have on a Java VM is not that big (6500 according to this answer). https://stackoverflow.com/questions/763579/how-many-threads-can-a-java-vm-support – Aayush Dwivedi Mar 22 '20 at 06:48
  • So set it lower. You can't actually execute thousands of GET requests simultaneously anyway. The network isn't multi-threaded. But the question you cited is eight years old. You need to do some experiments. – user207421 Mar 22 '20 at 06:57
  • https://github.com/AsyncHttpClient/async-http-client – Oleg Mar 22 '20 at 07:05
  • @Oleg That library also creates threads and pools. It is to be wondered whether it is truly any more asynchronous than what the OP is already doing. – user207421 Mar 22 '20 at 07:17
  • @user207421 It's only worker threads proportional to number of cpu cores. If that's not more asynchronous than creating a thread per request then that's just a meaningless word. – Oleg Mar 22 '20 at 07:42
  • @Oleg So, as I said, reduce it. Reduce it to the number of cores, and they are equivalent. – user207421 Mar 22 '20 at 08:45

1 Answers1

7

As @user207421 says:

  • A truly asynchronous (i.e. event driven) HTTP client application is complicated.
  • A multi-threaded (but fundamentally synchronous) HTTP client application is simpler, and scales to as many threads as you have memory for.

  • Assuming that you have 64 worker threads processing requests, the actual bottleneck is likely to be EITHER your physical network bandwidth, OR your available client-side CPU. If you have hit those limits, then:

    • increasing the number of worker threads is not going to help, and
    • switching to an asynchronous (event driven) model is not going to help.

  • A third possibility is that the bottleneck is server-side resource limits or rate limiting. In this scenario, increasing the client-side thread count might help, have no effect, or make the problem worse. It will depend on how the server is implemented, the nature of the requests, etc.

If your bottleneck really is the number of threads, then a simple thing to try is reducing the worker thread stack size so that you can run more of them. The default stack size is typically 1MB, and that is likely to be significantly more than it needs to be. (This will also reduce the ... erm ... memory overhead of idle threads, if that is a genuine issue.)

There are a few Java asynchronous HTTP client libraries around. But I have never used one and cannot recommend one. And like @user207421, I am not convinced that the effort of changing will actually pay off.

What can I [do] so that my threads don't sit idle waiting for the response?

Idle threads is actually not the problem. An idle thread is only using memory (and some possible secondary effects which probably don't matter here). Unless you are short of memory, it will make little difference.

Note: if there is something else for your client to do while a thread is waiting for a server response, the OS thread scheduler will switch to a different thread.

So my maximum number of simultaneous requests I can send out is limited by my thread [pool] size i.e. 64 here.

That is true. However, sending more simultaneous requests probably won't help. If the client-side threads are sitting idle, that probably means that the bottleneck is either the network, or something on the server side. If this is the case, adding more threads won't increase throughput. Instead individual requests will take (on average) longer, and throughput will stay the same ... or possibly drop if the server starts dropping requests from its request queue.

Finally, if you are worried of the overheads of a large pool of worker threads sitting idle (waiting for the next task to do), use an execution service or connection pool that can shrink and grow its thread pool to meet changing workloads.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216