3

I am using Mechanical Turk with Boto3 SDK.

As per the general documentation https://docs.aws.amazon.com/general/latest/gr/api-retries.html , “each AWS SDK implements exponential backoff algorithm” – so why do we need to implement it again in our code?

(I am also referring to AWS' answer here: https://forums.aws.amazon.com/thread.jspa?threadID=307015)

Gadam
  • 2,674
  • 8
  • 37
  • 56

1 Answers1

2

Default client back-off might not be enough for every use case.

I'm not familiar with this particular service client, but you generally can detect retries by using logging level logging.DEBUG. It will log retry attempts, so you can check how often and how many there are.

Some services have very specific rate limits in terms of N attempts in M time, so you can override default back-off by using botocore.config.Config property called retries and constructing service client while supplying config keyword.

Oleksii Donoha
  • 2,911
  • 10
  • 22
  • 1
    Can you clarify 'override default back-off by using botocore.config.Config property called retries'. I thought that config just helps you increase the number of retries, not anything to do with back off behavior? – Gadam Dec 14 '20 at 14:31
  • 2
    There are also different modes, take a look at https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html . Maybe adaptive mode is more suited to your use case. – Oleksii Donoha Dec 14 '20 at 15:20