12

I've been reading about TransferManager in the Amazon's AWS SDK for doing S3 uploads, the provided API allows for non-blocking usage, however it's unclear to me if the underlying implementation actually does asynchronous I/O.

I did some reading on the source-code of TransferManager and I cannot understand if the threads in the provided ExecutorService are being blocked or not.

My problem is that if this manager actually does asynchronous I/O without blocking that executor, then I could use the application's global thread-pool that is meant for CPU-bound stuff. So is this actually doing asynchronous I/O or not?

Alexandru Nedelcu
  • 8,061
  • 2
  • 34
  • 39

1 Answers1

14

After profiling and trying to understand the SDK's source-code I have come to the conclusion that yes, TransferManager does not work asynchronously, because it piggybacks on AmazonS3Client.putObject and such calls, while not blocking the threads per se, go in a loop until the http requests are finished, thus preventing progress in processing the thread-pool's queue.

Alexandru Nedelcu
  • 8,061
  • 2
  • 34
  • 39
  • 1
    This post on the AWS blog suggests otherwise: https://java.awsblog.com/post/Tx2Q9SGR6OKSVYX/Amazon-S3-TransferManager – DGolberg Sep 30 '14 at 14:54
  • 7
    @DGolberg no, it's not, that blog post doesn't claim otherwise and besides I've got proof after several hours of checking out the source code and doing profiling. The API of TransferManager is non-blocking, in the sense that it's offloading work to a configured thread-pool, but the threads in that thread-pool are blocking and hence unavailable for doing anything else. I'll probably write an article about it. Thanks for the down-vote, wasn't necessary. – Alexandru Nedelcu Sep 30 '14 at 18:36
  • 1
    @AlexandruNedelcu Indeed I believe you are correct, it's asynchronous in the sense that if you are using it then you can continue doing other things in whatever thread you made the API call, but the underlying implementation just makes another thread to do the work. Have an upvote =D – Richard Fung Sep 30 '14 at 20:19
  • 1
    @AlexandruNedelcu I guess I interpreted your question incorrectly; I apologize for that. Use of the transfer manager on the current thread is asynchronous, but no, the underlying threads are not, as you've stated. I need to remember to stop trying to answer questions when hindered by a lack of sleep... – DGolberg Sep 30 '14 at 20:55
  • 1
    @DGolberg that's OK, I asked myself this question because I was wondering in an application whether I should use the global and limited thread-pool that the app has configured for CPU-bound tasks or not. It would have been cool for efficiency reasons, but we've got a thread-pool meant for blocking I/O stuff anyway and the TransferManager is still pretty cool because it can batch multiple uploads together. – Alexandru Nedelcu Oct 01 '14 at 07:02
  • Ahh, I was kind of curious as to the reason for the question after I re-read it. I originally thought you were worried about being unable to do other work on the main thread calling the transferManager; something I've been messing around with a lot and finding more and more interesting. I've also recently found (and fixed) many mistakes I'd made previously, in how I was using it, so I guess I had a little bit of an ego as well, and rushed to answer without properly understanding the question (bad habit of mine), lol. Being tired didn't help matters. Glad you were able to find your answer! – DGolberg Oct 01 '14 at 08:29