2

In our app, we have attachments that are supposed to be secured. These attachments are stored in DO Spaces and we are generating resigned URLs to allow secured access.

The logic of signing the URLs looks like this:

String getSignUrl(String objectKey, Long expirationMillis) {
        String signedUrl = null;
        try {
            AmazonS3 s3Client = awsClientProviderService.getS3Client()

            // Set the presigned URL to expire after one hour.
            Date expiration = new Date()
            Long expTimeMillis = expiration.getTime();
            expTimeMillis += expirationMillis;
            expiration.setTime(expTimeMillis);

            GeneratePresignedUrlRequest generatePresignedUrlRequest =
                    new GeneratePresignedUrlRequest(spaceName, objectKey)
                            .withMethod(HttpMethod.GET)
                            .withExpiration(expiration);
            URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);

            signedUrl = url.toString()
        } catch (AmazonServiceException e) {
            log.error(e.message, e)
        } catch (SdkClientException e) {
            log.error(e.message, e)
        }

        signedUrl
    }

Where getS3Client is:

AmazonS3 getS3Client() {
        if (!s3Client) {
            AWSStaticCredentialsProvider credentials =
                    new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secret))
            s3Client = AmazonS3ClientBuilder
                    .standard()
                    .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(endpoint, region))
                    .withCredentials(credentials)
                    .build()
        }

        s3Client
    }

All good until now and works great.

However, when we start signing the URLs, after 50 of these requests we are getting the following error:

Unable to execute HTTP request: Timeout waiting for connection from pool

Digging around a bit, I tried adding maxConnections to the client: .withClientConfiguration(new ClientConfiguration().withMaxConnections(20))

This caused to get the error after 20 such requests. By the way, even if we sign 5 URLs first, then wait for 10 min, try another batch. The error occurs after 20 requests collectively.

Tried, 100 connections. Same story after 100 requests.

This made me think that the connections are not getting released for some reason. And the only thing that I could find is people proposing to increase maxConnections to 1000. But, that would just defer the issue.

What is the solution here? How can we release connections of S3Client?

Shurik Agulyansky
  • 2,607
  • 2
  • 34
  • 76
  • http GET is stateless , should it have state.. Does the app have service allowance limits. Based on time or quantity. If it is free it could have such constraints. NB you can simply throw your expiry time long addition straight into the Date(timestamp+seconds) constructor – Samuel Marchant Sep 25 '22 at 09:40

1 Answers1

0

Generally, "Timeout waiting for connection from connection pool" comes when all the connections in the pool are 'busy' and none is available for new request.

Check this question.

Depending on which version you are using, you can try using the 'shutdown' method, if there is one available for your version.

https://stackoverflow.com/questions/26866739/how-do-i-close-an-aws-s3-client-connection#:~:text=You%20don't%20need%20to,to%20S3%20when%20using%20AmazonS3Client.

Amol Limaye
  • 108
  • 1
  • 4