Uploading files to S3 in Multithreading

Question

I'm uploading files to s3 using the s3 AWS client in my spring application, but sometimes I'm getting the error,

com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool

So as a solution to this I used the following approach and it worked for now.

@Bean
public AmazonS3 s3Client() {
    return AmazonS3ClientBuilder
            .standard()
            .withClientConfiguration(new ClientConfiguration()
                    .withMaxConnections(100)
                    .withConnectionTimeout(100)
                    .withMaxErrorRetry(5))
            .build();
}

public String uploadFile() {
    // upload code
}

I have created this as a Spring Bean. But I am using this in a multithreading environment. So there will many concurrent requests at the same time. I see that AmazonS3ClientBuilder is annotated with @NotThreadSafe. So I need to know is it okay to use this as a bean in multithreading or else shall I use the above block of code inside the same uploadFile method? Can anyone explain me the best way? Thank you

score 2 · Accepted Answer · answered Sep 14 '21 at 11:23

You haven't share the actual code for the uploading procedure but I think your problems lies with that one. So, to answer your questions:

Regarding the @NotThreadSafe, you should not be worrying about this. What you use the builder for, is to create an instance of AmazonS3 client. This process is done during Spring initialization which in turn means the the whole process is handled by a single thread and thus is not affected by potential synchronization problems. Note that what this mentioned before is in regards to the AmazonS3ClientBuilder. The created AmazonS3Client object (created by invoking AmazonS3ClientBuilder#build) is marked as thread safe as you can see in the related source.
Regarding the problem you are experiencing. Unfortunately, without sharing the upload object logic there is no concrete way to understand the exact reason for it. However, I think that your problem stems from the fact that you are creating a number of concurrent upload requests that is higher than the configured max connection number. This is in turn will cause those requests to block, waiting for a connection from the http pool to be retrieved. If one cannot be retrieved in a timely manner, then the request will timeout (what you are experiencing). For more you can check the related issue here, which outlines the same behavior you are getting, albeit for a download action.

Uploading files to S3 in Multithreading

1 Answers1