2

I use Async Http Client library for download content to my server. My code looks like this:

AsyncHttpClient client = Dsl.asyncHttpClient();
FileChannel channel = new FileOutputStream("path_to_download").getChannel();

client.prepareGet("file).execute(new AsyncCompletionHandler<FileChannel>() {

    @Override
    public State onBodyPartReceived(HttpResponseBodyPart bodyPart) 
      throws Exception {
        channel.write(bodyPart.getBodyByteBuffer()); //blocking operation
        return State.CONTINUE;
    }

    @Override
    public FileChannel onCompleted(Response response) 
      throws Exception {
        channel.close();
        return channel;
    }
});

The problem is that in the onBodyPartReceived method. Since it is called several times and I write the incoming bytes to the FileChannel, at this point the thread is blocked.

There is a solution with AsynchronousFileChannel, but I looked at the source code of the main implementation of SimpleAsynchronousFileChannelImpl and ther in fact, the same thing happens, bytes writes in separate thread from Java cached thread pool. So this solution is async, but is blocking, cause it use for each write new thread.

Is there a way to write bytes to a file without creating unnecessary threads and not blocking the current thread?

UPDATE

I'll try to explain what I mean. AsyncHttpClient has as many threads as there are processor cores available. Suppose this number is 8. When data arrives to the onBodyPartReceived method, this code is conditionally speaking executed in the async-thread-3 thread. When I try to write incoming data to FileChannel, this thread is blocked and can no longer receive data. Therefore, I looked that there is a solution with AsynchronousFileChannel. But it has only one implementation - SimpleAsynchronousFileChannelImpl, which simply delegates write operation to the thread in cachedThreadPool. And this thread of cachedThreadPool is blocked, even though I released an async-3-thread.

So if I want to download 1000 files from the network and save it to disk, i need only 8 threads to read the data over the network, but for disk write I need 1000 threads. that is clearly ineffective. (although my disc is certainly faster than getting data over the network)

Peter Kozlovsky
  • 633
  • 2
  • 10
  • 28
  • 1
    *"not blocking the current thread"* You said it yourself, *"bytes writes in **separate** thread"*, which means that it is **not** in fact blocking the *current* thread. --- *"without creating unnecessary threads"* It would still need to "create a thread" to invoke the call-back method, so which *unnecessary* thread are you referring to? – Andreas May 29 '19 at 22:52
  • @Andreas The solution that is used now with FileChannel blocks the current thread. AsynchronousFileChannel is async blocking solution, because under the hood it use same code as FileChannel, but just does it in a separate thread from cached thread pool. 1 write operation - 1 thread, and this thread will be idle while writing and after call my callback method. If I have 200 write operations, 200 threads will be created, but a real non-blocking solution can use as little as 1 thread for several operations at a time. – Peter Kozlovsky May 30 '19 at 08:11
  • @PeterKozlovsky Did you ever manage to resolve this? I'm asking the exact same question myself - I have a similar scenario with AHC and `onBodyPartReceived` blocked by a blocking IO API – shays10 Jun 16 '20 at 14:13
  • @shays10 https://stackoverflow.com/questions/56397766/java-asynchttpclient-broken-file-while-writing-from-lazyresponsebodypart-to-asy maybe this can help you. As I understand it, in Linux it is impossible to read / write files in a non-blocking way, anyway, some thread will be in a blocked state. That's how the SimpleAsynchronousFileChannelImpl works. In linux 5.2 io_uring was added for asynchronous reading and writing, as I understand it, but this is not implemented in Java. – Peter Kozlovsky Jun 16 '20 at 14:34

0 Answers0