3

When I try to publish about a 1000 messages to a SQS FIFO queue (in a goroutine) using the SQS SDK, I get this error - operation error SQS: SendMessage, failed to get rate limit token, retry quota exceeded, 2 available, 5 requested

This is how I publish:

wg := sync.WaitGroup{}
messages := []string{}
for _, message := range messages {
        wg.Add(1)
        go func() {
            defer wg.Done()
            //publish to SQS
        }()
}
wg.Wait()

I try to fix by adding these to the aws config, following this suggestion - https://github.com/aws/aws-sdk-go-v2/issues/543#issuecomment-620124268:

"github.com/aws/aws-sdk-go-v2/config"

type NoOpRateLimit struct{}

func (NoOpRateLimit) AddTokens(uint) error { return nil }
func (NoOpRateLimit) GetToken(context.Context, uint) (func() error, error) {
    return noOpToken, nil
}
func noOpToken() error { return nil }

return config.LoadDefaultConfig(
        ctx,
        config.WithRetryer(func() aws.Retryer {
            return retry.NewStandard(func(options *retry.StandardOptions) {
                options.RateLimiter = NoOpRateLimit{}
            })
        }),
    )

But the failure rate even increased. How can I fix this please?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
shekwo
  • 1,411
  • 1
  • 20
  • 50
  • Have you tried adjusting the delay between requests to check how it affects your failure rate? And, finding the exact rate of limiting or ideal delay between requests by changing the delays? – Jishan Shaikh Apr 21 '23 at 07:50
  • If you don't really know what or how is it failing, you should enable logging in the AWS SDK for Go. Example: ```import ( "log" "os" "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/config" ) func main() { cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithClientLogMode(aws.LogRequestWithBody | aws.LogResponseWithBody), config.WithLogger(aws.NewDefaultLogger(log.New(os.Stdout, "", log.LstdFlags))), ) if err != nil { log.Fatalf("failed to load config: %v", err) } }```. – Jishan Shaikh Apr 21 '23 at 07:52

0 Answers0