Stack Overflow
Stack Overflow

Questions tagged [amazon-sqs]

Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable, hosted queue for storing messages as they travel between computers.

Resources

Tutorial

4134 questions
717
votes
9 answers

What is the difference between Amazon SNS and Amazon SQS?

When would I use SNS versus SQS, and why are they always coupled together?
Nick Ginanto
  • 31,090
  • 47
  • 134
  • 244
224
votes
13 answers

Why should I use Amazon Kinesis and not SNS-SQS?

I have a use case where there will be stream of data coming and I cannot consume it at the same pace and need a buffer. This can be solved using an SNS-SQS queue. I came to know the Kinesis solves the same purpose, so what is the difference? Why…
Apoorv
  • 2,525
  • 2
  • 14
  • 18
139
votes
14 answers

Best way to move messages off DLQ in Amazon SQS?

What is the best practice to move messages from a dead letter queue back to the original queue in Amazon SQS? Would it be Get message from DLQ Write message to queue Delete message from DLQ Or is there a simpler way? Also, will AWS eventually…
Matt Dell
  • 9,205
  • 11
  • 41
  • 58
135
votes
4 answers

SQS vs RabbitMQ

I need to create a queue for processing. The queue itself is relatively low-volume. There might be about 1,000 writes to it per hour. The execution of each task might take about a minute each, and are processed almost as soon as the item is added to…
David542
  • 104,438
  • 178
  • 489
  • 842
95
votes
3 answers

Kafka or SNS or something else?

Sorry if it is a newbie question. But I'm trying to understand what should I use. As far as I understand Kafka is : Apache Kafka is a distributed publish-subscribe messaging system. And SNS is also pub/sub system. My goal is to use some queue…
Vor
  • 33,215
  • 43
  • 135
  • 193
92
votes
2 answers

Why do SqS messages sometimes remain in-flight on queue

I'm using Amazon SQS queues in a very simple way. Usually, messages are written and immediately visible and read. Occasionally, a message is written, and remains In-Flight(Not Visible) on the queue for several minutes. I can see it from the console.…
Jerico Sandhorn
  • 1,880
  • 1
  • 18
  • 24
87
votes
6 answers

How to prevent duplicate SQS Messages?

What is the best way to prevent duplicate messages in Amazon SQS? I have a SQS of domains waiting to be crawled. before I add a new domain to the SQS I can check with the saved data to see if it has been crawled recently, to prevent duplicates. The…
Marcus Lind
  • 10,374
  • 7
  • 58
  • 112
86
votes
6 answers

How to turn off the pager for AWS CLI return value?

I am attempting to utilize the AWS CLI along with a for loop in bash to iteratively purge multiple SQS message queues. The bash script works almost as intended, the problem I am having is with the return value each time the AWS CLI sends a request.…
Colin Armstrong
  • 1,609
  • 1
  • 12
  • 18
84
votes
7 answers

Emulating Amazon SQS during development

I'm quite interested in beginning some development using Amazon SQS, perhaps SimpleDB too, my question is this, are there any open source solutions that mimic the functionality, just for the purposes of development. I've already encountered the…
Derek Mortimer
  • 1,061
  • 1
  • 8
  • 11
79
votes
6 answers

SNS to Lambda vs SNS to SQS to Lambda

I'm trying to understand whether I need SQS in my workflow if someone can help explain. In my app, when an action is taken, it submits info to SNS topic which invokes Lambda to do some processing. This is working great as it is. When I do research…
Saumil Shah
  • 1,383
  • 2
  • 11
  • 20
67
votes
4 answers

How to peek at messages in the queue

I don't want the message to count as "read" but I'd like to know what's in the queue. The documentation: http://boto.s3.amazonaws.com/ref/sqs.html#module-boto.sqs Isn't very straight forward about what absorbs a message and what doesn't. The dump…
Ralphleon
  • 3,968
  • 5
  • 32
  • 34
64
votes
3 answers

Using Amazon SQS with multiple consumers

I have a service-based application that uses Amazon SQS with multiple queues and multiple consumers. I am doing this so that I can implement an event-based architecture and decouple all the services, where the different services react to changes in…
user2205763
  • 1,499
  • 2
  • 14
  • 30
61
votes
8 answers

How to process SQS queue with lambda function (not via scheduled events)?

Here is the simplified scheme I am trying to make work: http requests --> (Gateway API + lambda A) --> SQS --> (lambda B ?????) --> DynamoDB So it should work as shown: data coming from many http requests (up to 500 per second, for example) is…
xtx
  • 4,306
  • 4
  • 28
  • 30
58
votes
7 answers

Celery with Amazon SQS

I want to use Amazon SQS as broker backed of Celery. There’s the SQS transport implementation for Kombu, which Celery depends on. However there is not enough documentation for using it, so I cannot find how to configure SQS on Celery. Is there…
minhee
  • 5,688
  • 5
  • 43
  • 81
58
votes
10 answers

SNS topic not publishing to SQS

I am trying to prototype a distributed application using SNS and SQS.I have this topic: arn:aws:sns:us-east-1:574008783416:us-east-1-live-auction and this queue: arn:aws:sqs:us-east-1:574008783416:queue4 I created the queue using the JS Scratchpad.…
user427875
  • 1,245
  • 2
  • 11
  • 16
1
2 3
…
99 100