2

FIFO queue guarantees exactly-once processing. If a message visibility timeout is less than the processing time, when the visibility timeout expires, the message won't be deleted. Since it's FIFO queue, I suppose the message won't be processed by other consumers again. So what is need for visibility timeout in FIFO queue?

eriee
  • 416
  • 2
  • 15
  • In the case of the message that didn't complete processing within the visibility timeout, the message is not considered to have been processed and is returned to the queue, at the head of its message group. More here: https://tomgregory.com/3-surprising-facts-about-aws-sqs-fifo-queues – jarmod Feb 22 '22 at 01:19
  • @jarmod If i understand it correctly, a single message can actually be processed multiple times. It's also possible that multiple consumers can process the same message concurrently (consumer A not finish yet while consumer B start processing). emm, it doesn't meet my expectation for the FIFO queue. – eriee Feb 22 '22 at 01:30
  • No, that's not it. You need to have a consistent definition of what "processed" means first of all. It doesn't mean "started processing but we don't know what happened subsequently". It means the app receiving the message knows that it completed its processing and it deleted the message, thereby unblocking the queue. If you want it the other way as in "when I receive the message consider it processed" then you have to implement that yourself (by simply deleting the message from the queue). Nothing else makes sense. Also, no, multiple consumers will not see the same message. B is blocked. – jarmod Feb 22 '22 at 01:33
  • Why is B blocked? First A gets the message, processes it half way (retrieves the data but not delete it because it wants to guarantee FIFO), and then the message becomes visible. Who is going to get the message again while A is still busy processing? If no one, then the message is still processed once by A. – eriee Feb 22 '22 at 01:51
  • OK, I see where you're going. The message group is blocked for the duration of the visibility timeout or until the message is deleted. If A receives the message but fails to delete it within the visibility timeout then the message will be visible again and can be consumed by B. In this case A has failed to process the message (it did not delete it within the visibility timeout). Because A failed, by that definition, the message will be visible again and processed by the next consumer. You need to be careful how you configure the timeout relative to the time your code takes to process it. – jarmod Feb 22 '22 at 02:01
  • I see. A bad visibility timeout configuration can actually allow a message to be received multiple times. It's possible A and B both processed the data from the message. I need to make the processing idempotent. – eriee Feb 22 '22 at 02:08
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/242251/discussion-between-jarmod-and-eriee). – jarmod Feb 22 '22 at 02:17

1 Answers1

2

Entire FIFO queue (within a message group) will stop being processed unless you delete the message or it gets visible again. From docs:

When you receive a message with a message group ID, no more messages for the same message group ID are returned unless you delete the message or it becomes visible.

So your single message which hasn't been deleted from the queue, for whatever reason, will block the entire message group in the queue.

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • When visibility timeout expires, it becomes visible. So based on the "OR" condition, the message will be processed again? – eriee Feb 22 '22 at 01:26
  • @eriee Yes, until you delete it or have DLQ setup to automatically remove faulty msgs. – Marcin Feb 22 '22 at 01:27