79

I'm trying to understand whether I need SQS in my workflow if someone can help explain. In my app, when an action is taken, it submits info to SNS topic which invokes Lambda to do some processing. This is working great as it is.

When I do research online, it seems that people are using SQS in this stack as well where SNS would put info on SQS and then SQS would then invoke Lambda.

I guess what I'm trying to understand is the need for SQS in this. What value does that add? In other words, what am I losing by invoking my Lambda directly from SNS?

BinaryButterfly
  • 18,137
  • 13
  • 50
  • 91
Saumil Shah
  • 1,383
  • 2
  • 11
  • 20

6 Answers6

78

Primary advantage of having a SQS in between SNS and Lambda is Reprocessing. Assume that the Lambda fails to process certain event for some reason (e.g. timeout or lack of memory footprint), you can increase the timeout (to max 5 minutes) or memory (to max of 1.5GB) and restart your polling and you can reprocess the older events.

This would not be possible in case of SNS to Lambda, wherein if Lambda fails the event is lost. And even if you configure DLQ you would still have to make provisions for reading that separately and processing the message

So if your events are critical and you don't want to miss out on them, then go for SNS - SQS - Lambda

The other advantage of having SQS is cost saving on Lambda invocations (Thanks @codesinthedark for bringing this up). You can have much better scaling and less cost, as it allows you to process messages in batches. So one lambda can be executed for a batch of 10 messages while in case of direct SNS each message would trigger a lambda invocation.

Arafat Nalkhande
  • 11,078
  • 9
  • 39
  • 63
  • 1
    Repeating as a comment here instead of as a different answer below... [AWS Lambda Adds Amazon Simple Queue Service to Supported Event Sources](https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/) – dmulter Jul 02 '18 at 22:11
  • Right, @dmulter, SQS triggering Lambda support was added on 28 JUN 2018. However, the question mentions that "In my app, when an action is taken, it submits info to SNS topic which invokes LAMBDA to do some processing", so this new feature would still not answer the question. – Arafat Nalkhande Jul 03 '18 at 05:10
  • 1
    Any Lambda function invoked asynchronously is retried twice and then can be sent to a DLQ which can be a SNS or SQS https://docs.aws.amazon.com/lambda/latest/dg/dlq.html So the reason given in this answer as to why add SQS between SNS and lambda does not make sense – user3311298 Jul 17 '18 at 18:45
  • 2
    I do not agree that this post does not answer the question, because assume that you have configured a DLQ instead of going the direct route of adopting SQS, then for all the failures you will have to go to the DLQ. If for some reason the failures were simply timeouts or due to insufficient memory footprints then I'll have to make separate provisions to read the DLQ and process the messages, whereas if the SQS was used in between, then I simply have to reconfigure the lambda with larger memory footprint and rest if taken care off automatically – Arafat Nalkhande Jul 30 '18 at 08:27
  • Thanks for pointing out the DLQ. I've made slight adjustments to the answer and added that part – Arafat Nalkhande Jul 30 '18 at 08:29
  • This is assuming that SQS never goes down, otherwise events can still get lost. But I guess that's a reasonably safe assumption? – lfk May 15 '19 at 04:29
  • Lambda doesn't "fail" easily and "lose" the data if there is no external component such as http/s microservice, email, or sms is involved. More info here: https://aws.amazon.com/lambda/faqs/#Scalability_and_availability – Payam Nov 20 '20 at 19:08
  • @Payam, The link that you posted says "On failure, Lambda functions being invoked synchronously will respond with an exception. Lambda functions being invoked asynchronously are retried at least 3 times"... What if it fails even after 3 times – Arafat Nalkhande Dec 02 '20 at 04:02
  • @ArafatNalkhande, If the process needs high reliability, then the Lambda should forward the results to another AWS service. For example, we use SNS->Lambda->Elasticache/Redis a lot. Amazon complies with Six-Sigma (99.9999% Availability), and you can increase the number of retires or eventually forward the failed processes to a DLQ SQS – Payam Dec 03 '20 at 23:09
  • Lets say I forward that to Redis I would then need to add some compute tier to process it from Redis and I would assume the SNS to SQS stuff will serve the same purpose and would be much simpler approach – Arafat Nalkhande Dec 04 '20 at 01:29
  • 5
    I think that you forgot one huge benefit of using lambda with SQS instead of SNS. You can have much better scaling and less cost, as it allows you to process messages in batches so one lambda can be executed for 10 messages while each SNS message would trigger lambda and it would fail or throttle for over 1000msg/s. You would pay one lambda execution and one SQS request instead of 10 requests for SNS. You can update your answer. – CodesInTheDark Aug 17 '21 at 12:49
  • @CodesInTheDark But an SQS request is more expensive than a Lambda invocation and I expect Lambda can do a lot more than 1000 msg/s if each execution takes less than a second. But, agree, too much concurrency could be a problem for a variety of reason. – user1055568 May 12 '23 at 15:50
  • @user1055568 Delivery from SQS to Labda is free. – CodesInTheDark May 17 '23 at 12:59
  • Right, but SQS request costs $.24/million. If you can replace SNS with SQS, there is big savings, but question was about inserting an SQS. – user1055568 May 17 '23 at 14:37
9

I think couple of things changed in 2019 and SQS can trigger lambda via event source mapping which is mentioned by @alexs. Related blog post: https://aws.amazon.com/about-aws/whats-new/2018/04/aws-lambda-now-supports-amazon-sqs-as-event-source/

To summarize, you can use SQS to lambda with following benefits:

  • Reprocessing events in case of failure and configure how many times a message should be retried before you give up (receive count)
  • Longer retention period
  • Typically chosen in the scenarios where there is a long running job and the lambda polls one by one from job queue.

you can choose to use SNS:

  • If you need to fanout a single message to multiple destinations, say X message should be processed by Y and Z applications. I feel this is the biggest advantage, and if you want reliability in this, you can couple SNS and SQS together.
  • You do not care about lost messages. Remember that there are still retry stratergies when using SNS(linear, geometric, exponential etc)
  • Typically used in the cases where you can ingest/process messages faster. This can sometimes be a problem as well; imagine a scenario where there is a SNS notification for every email that your business receives and you dont have enough lambda concurrency to process all of them. You can solve this by putting an SQS to consume at your own pace.

In both the cases, there can be duplicate messages(in the cases of retry) and there cannot be order guarantees. If you need one, consider Kinesis streams.

St1id3r
  • 313
  • 5
  • 12
  • 2
    Things changed again in 2020 with SNS FIFO - providing strict message ordering and deduplicated message delivery: https://aws.amazon.com/blogs/aws/introducing-amazon-sns-fifo-first-in-first-out-pub-sub-messaging/ – Jacek Kościesza Oct 25 '20 at 09:28
  • The retry policy for SNS only applies to situations where the message is not received because the consumer's service is unavailable. If your Lambda fails to process the message correctly, it can be configured to send its message to a DLQ (after 2 retries by default), but it will never be sent back to SNS for re-transmission. – Scott Simontis Feb 25 '22 at 15:41
6

You can now use SQS as en event source

AWS Lambda Adds Amazon Simple Queue Service to Supported Event Sources

alexs
  • 1,507
  • 18
  • 17
5

SQS does not invoke Lambda. SQS cannot invoke anything. People using Lambda with SQS are running Lambda on an event timer, like once a minute, and every time the function runs it polls SQS to see if there is a message to process.

If you don't need to queue things up and prevent too many Lambda functions from running concurrently then you don't need a queue system like SQS.

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • Sorry. I guess what I meant to say is that LAMBDA gets invoked automatically when the connected SNS TOPIC receives any info. This has been working great. I guess what I'm trying to ask is would this approach scale? Say for instance, there are 100s of users trying to perform that action, would having an SQS in that case help with anything or would my current approach work just fine? – Saumil Shah Mar 08 '17 at 18:00
  • 2
    It entirely depends on what your Lambda function is doing. If it is inserting into a relational database that has a limit on connections, or hitting a third party API that has rate limits, then you will run into issues. If it is just updating a file in S3 or something, then you probably won't run into issues. You might have to apply for an increase to the allowed concurrent Lambda functions in your AWS account, but that would be about it. – Mark B Mar 08 '17 at 18:03
  • It looks like SQS can trigger lambda now, what would your updated advice be in light of that new capability? – Davos Oct 16 '18 at 00:43
  • 1
    @Davos Adding SQS would allow you to control the concurrency and retries, so go with SQS if you want control over those things. It would add a bit more cost though. – Mark B Oct 16 '18 at 01:53
  • @Davos yes SQS has a lambda trigger now but internally lambda polls the SQS. so you don't have to manually trigger event on lambda to poll message. – Vishal Patel Oct 01 '20 at 16:13
5

Adding to @Arafat Nalkhande's answer here are few benefits of SQS's lambda

  1. In SQS we can put a delay, so that message gets processed after some time, it may be useful in the scenario where data takes time to be available.

  2. SQS can serve as a contingency store, lets say downstream services are unavailable, message can be retained in sqs for 15 days.

best wishes
  • 5,789
  • 1
  • 34
  • 59
0

Depends on how you want to handle retries, error handling. Every AWS resource have a way to retry failed events so you would want to learn how SNS handles failed events, can it handle it or not and if not you use SQS. But you can always push failed events from lambda back to SNS so you can process them again. But also consider if you really want SQS, that you truly want a queue service and not just for retries because you can always use other ways to handle such issues.

Ankur Kothari
  • 822
  • 9
  • 11