Questions tagged [aws-sqs-fifo]
100 questions
8
votes
1 answer
Making SQS message visible again using partial batch response
Assuming:
there is AWS Lambda that processes messages in 10-elements batches from AWS SQS FIFO queue with 25 available message group ids (assign in a random fashion)
the processing of a single message takes ~30-60 sec
SQS message visibility timeout…
Mark
- 275
- 1
- 4
- 16
6
votes
1 answer
Understanding AWS FIFO Queue behaviour
Was playing around with AWS SQS FIFO Queue locally in localstack with AWS Java sdk v2 & Spring Boot.
I created one endpoint to send messages through one publisher and three endpoints to receive/poll messages from queue via three consumers in Spring…
Tahniat Ashraf
- 1,020
- 2
- 12
- 21
4
votes
3 answers
Eventbridge - Use FIFO SQS for deduplication
I need some events to be delivered exactly once, but I have no control of the message processor (so I can‘t make the recipient idempotent).
Is it possible to route events from Eventbridge to a FIFO SQS for deduplication and from the FIFO sqs to the…
florian norbert bepunkt
- 2,099
- 1
- 21
- 32
3
votes
0 answers
Rate limiting errors publishing to SQS
When I try to publish about a 1000 messages to a SQS FIFO queue (in a goroutine) using the SQS SDK, I get this error - operation error SQS: SendMessage, failed to get rate limit token, retry quota exceeded, 2 available, 5 requested
This is how I…
shekwo
- 1,411
- 1
- 20
- 50
3
votes
0 answers
Lambda with 1 reservedConcurrency getting throttled even when FIFO queue(trigger event) has same messageGroupId across all messages
I have a SAM stack with 1 lambda, 1 SQS and 1 DLQ. The message in SQS acts as an event source for the lambda. The lambda has ReservedConcurrentExecutions of 1. The batch size of event(from SQS to lambda) is also 1. The timeout of lambda is 300…
Nandita Sharma
- 13,287
- 2
- 22
- 35
3
votes
0 answers
AWS SQS DLQ messages become not visible without consumer/trigger
In the architecture I'm working on I have a simple SQS queue with a DLQ.
I have noticed a specific behaviour that doesn't make sense in my mind on the DLQ: even if my DLQ has NO lambda trigger it looks like the messages can become not visible…
Massimo Polimeni
- 4,826
- 4
- 27
- 54
3
votes
0 answers
Decrease throughput of AWS lambda with SQS trigger
Our system differentiates between what we're calling 'single events' and 'bulk events'. Single events are the result of user interactions and arrive just a few at a time, now and then. Bulk events are the result of actions by an admin and can…
dhollinden
- 43
- 3
2
votes
1 answer
Same Message Group Tasks executing in parallel in Celery with Amazon SQS FIFO
I have a setup where I'm using Celery as the task queue with Amazon SQS FIFO. My goal is to ensure sequential processing of tasks within the same message group ID, while allowing tasks with different message group IDs to be processed in parallel.…
Manish Patel
- 23
- 1
- 6
2
votes
1 answer
AWS SQS.FIFO reads the first 20,000 messages to determine message groups, what is the order of reading?
What mechanism does SQS.FIFO use to read the first 20,000 messages?
Here's an example for some context:
On a FIFO queue, we have message groups:
| Group | Number of messages |
| A | 50,000 |
| B | 100 |
| C …
nnedoklanov
- 301
- 2
- 7
2
votes
1 answer
AWS API Gateway with FIFO SQS
How to integrate AWS API Gateway with FIFO SQS?
It is easy to integrate API Gateway with standard SQS with the template. But for FIFO queue, we need to pass groupId and deduplicationId.
Brand
- 65
- 4
2
votes
2 answers
SQS FIFO Listener Receives Messages Before Previous One Finished
I have a spring-boot application with aws sqs fifo, which need to process messages in fifo manner. Lets say there are 3 messages (m1,m2,m3) under same group id in fifo queue. order of execution should be m1,m2 and m3.
m2 should wait until m1(take…
kevin_Aberathna
- 441
- 1
- 5
- 13
2
votes
0 answers
Cascading after_update operations with sqlalchemy not working
We have a parent/child table and want to be able to propagate updates up from the children/leaves to the parent root.
For example, given we have a structure like
{
id: 1,
state: "in-progress",
children: [{
id: 2,
state: "in-progress",
…
Frank Siderio
- 36
- 1
2
votes
1 answer
What happens to duplicate messages sent during the deduplication interval in AWS FIFO Queues?
I have a FIFO queue with multiple consumers. The producer includes a Deduplication ID with every message.
In the docs, it says:
If a message with a particular message deduplication ID is sent successfully, any messages sent with the same message…
Rick
- 628
- 1
- 7
- 20
2
votes
1 answer
What will happen if message visibility timeout expires in SQS FIFO queue?
FIFO queue guarantees exactly-once processing. If a message visibility timeout is less than the processing time, when the visibility timeout expires, the message won't be deleted. Since it's FIFO queue, I suppose the message won't be processed by…
eriee
- 416
- 2
- 15
2
votes
2 answers
Does Amazon SQS FIFO provide message ordering?
I am trying to implement AWS SQS FIFO queue using spring boot(v2.2.6.RELEASE).
Created a queue, "Testing.fifo" in aws.
Left all other fields to default while creating the queue.
My producer and consumer to the queue run on a single service.
code to…
Abhijith V Nair
- 23
- 5