7

I have four current consumers listening to the same queue on Amazon AWS. When pulling message down from the queue, it appears sometimes the same message is consumed by two different consumers. Please see the log below:

18:01:46,515 [jmsContainer-2] DEBUG - Received message from the queue: ID:3698a927-930b-4d6a-aeca-f66922528792

18:02:12,825 [jmsContainer-3] DEBUG - Received message from the queue: ID:3698a927-930b-4d6a-aeca-f66922528792

I have a JMS container setup with 4 concurrent consumers. I have the visibility timeout set to 30s.

Since the message is received by container2, how come container3 is still able to access it?

Does JMS container do auto-acknowledgment before or after the execution of the listener method (handleMessage)?

Gary Xue
  • 143
  • 2
  • 9
  • Possible duplicate of [Using many consumers in SQS Queue](http://stackoverflow.com/questions/37472129/using-many-consumers-in-sqs-queue) – Krease Jul 10 '16 at 17:37

2 Answers2

9

Amazon does not guarantee exactly once delivery with SQS. They guarantee "at least once" delivery. This is addressed in the FAQ https://aws.amazon.com/sqs/faqs/

You have to keep this in mind and design your system to gracefully handle duplicate message delivery.

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • 1
    That's true, technically, but I find it to be *extremely* rare. Examples should be hard to find. The problem is usually not SQS but a user component resetting the visibility timeout. – Michael - sqlbot Mar 16 '16 at 01:46
  • 1
    Thank you for the explanation. This actually happens pretty frequently for me, and I don't have visibility timeout reset in my application code. – Gary Xue Mar 16 '16 at 15:26
  • If processing of the message takes longer than the invisibility timeout of the message, the message will reappear on the queue and be available for another worker to read it. This is the most common cause of duplicate messages. – garnaat Mar 25 '16 at 14:48
  • @garnaat, but by default, JMS container has auto-acknowledgement enabled, which means the message would be acknowledged right after the handle message method is invoked. If I'm correct, then SQS does not have to wait until the message is processed by the client before it can delete the massage. – Gary Xue Mar 28 '16 at 21:54
  • If by *acknowledged* you mean deleted from the queue, then you are correct. But then I sure hope the processing of the message doesn't fail because it won't appear back on the queue for re-processing. – garnaat Mar 28 '16 at 21:57
4

This is possible now with FIFO SQS queues. You can make sure exactly one client receives the message and then delete it while the message is in-flight

More information here:

https://aws.amazon.com/about-aws/whats-new/2016/11/amazon-sqs-introduces-fifo-queues-with-exactly-once-processing-and-lower-prices-for-standard-queues/

Mebin Joe
  • 2,172
  • 4
  • 16
  • 22
Jagadish Kamath
  • 103
  • 2
  • 8