Exponential apiCallAttemptTimeout in AWS SDK for Java V2

Question

I'm currently using the AWS SDK for Java v2 (specifically v2.17.1) and the S3AsyncClient::getObject method.

final GetObjectRequest getObjectRequest = GetObjectRequest.builder()
    .bucket("my bucket")
    .key("some key")
    .overrideConfiguration(AwsRequestOverrideConfiguration.builder()
        .apiCallAttemptTimeout(Duration.ofSeconds(1))
        .build())
    .build();

return s3AsyncClient
    .getObject(getObjectRequest, AsyncResponseTransformer.toBytes())
    .thenAppply(/* snipped for brevity */);

This mostly works fine, but occasionally I get bursts of these errors:

software.amazon.awssdk.core.exception.ApiCallAttemptTimeoutException: HTTP request execution did not complete before the specified timeout configuration: 1000 millis

I currently have the following configuration on the builder that creates that s3AsyncClient above:

S3AsyncClient.builder()
    // other options snipped for brevity
    .overrideConfiguration(ClientOverrideConfiguration.builder()
        .retryPolicy(RetryPolicy.builder()
            .numRetries(2)
            .build())
        .build())
    .build()

Is there a way to make it so that different apiCallAttemptTimeout values are used depending on how many retry attempts have been made? In other words, how do I get exponential retry timeouts working?

I'm aware a newer version of the SDK includes a new ADAPTIVE RetryMode option, but my understanding is that this only applies when you're being rate limited by AWS. I've also seen parts of the S3 code that can exponentially delay the time between retries, but not the timeout for each retry.

Any help is appreciated - thank you!

Hey Borja - does this answer help at all? https://stackoverflow.com/a/40370108/4800344 — Ermiya Eskandary, May 13 '22 at 11:51

score 0 · Answer 1 · answered May 13 '22 at 20:00

You're nearly there.

In AWS SDK v2, the concept of RetryPolicy & BackoffStrategy has been carried over from V1 though with some minor differences in terms of how to configure it.

The builder allows for some complex and unique configurations but with some digging into the documentation, you'll find a set of PredefinedBackoffStrategies with one of them being the ExponentialBackoffStrategy.

This predefined strategy allows you to set the base (initial) delay & then the max back-off duration. You can set this when creating your client.

This should help:

final int initialDelayInMilliSeconds = 1 * 1000;
final int maxDelayInMilliSeconds = 60 * 1000;

final PredefinedBackoffStrategies.ExponentialBackoffStrategy backoffStrategy =
        new PredefinedBackoffStrategies.ExponentialBackoffStrategy(
                initialDelayInMilliSeconds,
                maxDelayInMilliSeconds
        );

final RetryPolicy retryPolicy = RetryPolicy.builder()
        .backoffStrategy((BackoffStrategy) backoffStrategy)
        .build();

return S3AsyncClient.builder()
        // other options snipped for brevity
        .overrideConfiguration(ClientOverrideConfiguration.builder()
                .retryPolicy(retryPolicy)
                .build())
        .build();

Thanks for digging into this Ermiya! One thing to note is that ExponentialBackoffStrategy is a V1 class - see the V2 ones: https://github.com/aws/aws-sdk-java-v2/tree/master/core/sdk-core/src/main/java/software/amazon/awssdk/core/retry/backoff Either way, this is exponential backoff, not exponentially increasing the timeout on each attempt :( — Borja Canseco, May 13 '22 at 21:26
Ah you want to increase the timeout - why may I ask? Why can't you just set it to a large amount? That value is normally not modified. XY problem maybe? I don't think that's possible as it's not something that would be exponentially retried. It's a timeout for connecting to AWS. — Ermiya Eskandary, May 13 '22 at 21:32
Definitely possible it's an XY problem -- I'll look into recommended backoff strategies instead. Thank you! — Borja Canseco, May 13 '22 at 21:42
This is increasing the time between failures - it may well do what you're asking for. Try it out and let me know what is not as expectred — Ermiya Eskandary, May 13 '22 at 21:46

Exponential apiCallAttemptTimeout in AWS SDK for Java V2

1 Answers1