25

According to the documentation:

Q: How many times will I receive each message?

Amazon SQS is engineered to provide “at least once” delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies.

Is there any good practice to achieve the exactly-once delivery?

I was thinking about using the DynamoDB “Conditional Writes” as distributed locking mechanism but... any better idea?

Some reference to this topic:

Community
  • 1
  • 1
Filippo Vitale
  • 7,597
  • 3
  • 58
  • 64

4 Answers4

10

FIFO queues are now available and provide ordered, exactly once out of the box.

https://aws.amazon.com/sqs/faqs/#fifo-queues

Check your region for availability.

Sam Sippe
  • 3,160
  • 3
  • 27
  • 40
  • 4
    FIFO queues are not compatible with SNS , so if you are using a combination of SNS+SQS you can't use it. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-subscribe-queue-sns-topic.html – Gustavo Preciado Feb 14 '18 at 18:48
  • 3
    AWS FIFO queues provied exactly once ONLY IF all the parties (producer, queue and consumer) are single threaded and more.. I would suggest reading this article for more info - https://www.linkedin.com/pulse/truth-aws-sqs-fifo-harleen-mann/ – human Jul 22 '18 at 13:33
9

The best solution really depends on exactly how critical it is that you not perform the action suggested in the message more than once. For some actions such as deleting a file or resizing an image it doesn't really matter if it happens twice, so it is fine to do nothing. When it is more critical to not do the work a second time I use an identifier for each message (generated by the sender) and the receiver tracks dups by marking the ids as seen in memchachd. Fine for many things, but probably not if life or money depends on it, especially if there a multiple consumers.

Conditional writes sound like a clever solution, but it has me wondering if perhaps AWS isn't such a great solution for your problem if you need a bullet proof exactly-once solution.

Jeff
  • 6,646
  • 5
  • 27
  • 33
  • 6
    +1 I agree, Amazons trade offs for sqs downgrades it as a messaging solution for many types of applications. I wish they would offer their own sync solution as an optional behavior you could enable at an understood performance cost, instead of all of us having to labor over the same workarounds. – Jerico Sandhorn Nov 09 '13 at 15:34
4

Another alternative for distributed locking is Redis cluster, which can also be provisioned with AWS ElasticCache. Redis supports transactions which guarantee that concurrent calls will get executed in sequence.

One of the advantages of using cache is that you can set expiration timeouts, so if your message processing fails the lock will get timed release.

Sergey
  • 3,214
  • 5
  • 34
  • 47
1

In this blog post the usage of a low-latency control database like Amazon DynamoDB is also recommended: https://aws.amazon.com/blogs/compute/new-for-aws-lambda-sqs-fifo-as-an-event-source/

Amazon SQS FIFO queues ensure that the order of processing follows the message order within a message group. However, it does not guarantee only once delivery when used as a Lambda trigger. If only once delivery is important in your serverless application, it’s recommended to make your function idempotent. You could achieve this by tracking a unique attribute of the message using a scalable, low-latency control database like Amazon DynamoDB.

In short - we can put item or update item in dynamodb table with condition expretion attribute_not_exists(for put) or if_not_exists(for update), please check example here https://stackoverflow.com/a/55110463/9783262

If we get an exception during put/update operations, we have to return from a lambda without further processing, if not get it then process the message (https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-idempotent/)

The following resources were helpful for me too:

https://ably.com/blog/sqs-fifo-queues-message-ordering-and-exactly-once-processing-guaranteed

https://aws.amazon.com/blogs/aws/introducing-amazon-sns-fifo-first-in-first-out-pub-sub-messaging/

https://youtu.be/8zysQqxgj0I

SAndriy
  • 670
  • 2
  • 15
  • 25