0

I have an application to handle bounce messages received from AWS SQS through SES through SNS.

The logic is as follow

  1. Initialize Ms SQL Server connection pool
  2. Poll data from SQS with long polling enabled and Wait Time is set to 20 seconds and max number of messages received to set to 10
  3. If there are messages meaning the callback from "sqs.receiveMessage" method has been called
  4. using "async.each" to process message(s) in parallel
    • for each message :
      • log it to Sql Table
      • disable Account if hard bounce
      • delete from queue
  5. Repeat step 2

Because SQS doesn't work exactly like a pub/sub model like how Kafka Consumer/Producer works. I have to keep polling for data. Initially, I tried using while(true) {} loop but it doesn't work. I am thinking maybe the reason is because the main call stack always has something to do and it is never empty?

Index.ts

async.series([
    function start(step) {
        LOG.info('Initializing Ms SQL Server Connection Pool');
        sql = MssqlDao.getInstance();
        sql.init().then(() => {
            if (sql.isInitialised) {
                LOG.info('SQL Server Initialized');
            }
            step();
        });
    }, function listen() {
        const bounces = new Bounces(sql.connectionPool);

        **// THIS WON'T WORK. Why?**
        while(true) {
          LOG.info('Initializing Bounces listener continuously');
          bounces.listen();
        }

        **// It works if I use CronTab or remove while loop and use recursion**

    }
]);

Bounces.ts

listen() {

        const queueURL: string | undefined = process.env.QUEUE_URL;
        if (!queueURL) {
            throw new Error(`Queue url is required`);
        }
        const params: any = {
            AttributeNames: [
                "SentTimestamp"
            ],
            MaxNumberOfMessages: 10,
            MessageAttributeNames: [
                "All"
            ],
            QueueUrl: queueURL,
            //VisibilityTimeout: 20,
            WaitTimeSeconds: 20
        };
        let that = this as any;

        sqs.receiveMessage(params, function (err, data: ReceiveMessageResult) {

            // THIS CALLBACK NEVER GETS CALLED INSIDE WHILE LOOP

            if (err) {
                LOG.error("Receive Error", err);
            } else if (data.Messages) {

                let messages = data.Messages;

                async.each(messages, async (_message: any) => {
                    let message: any = JSON.parse(_message.Body);
                    // Insert
                    let parameters: Parameter = {...};

                    const parameterKeys: string[] = Object.keys(parameters);

                    try {

                        let request = that.sql.request() as Request;
                        await that.insert(parameterKeys, request, parameters);

                        let eventStatusCode = parameters.eventStatusCode;
                        let email = parameters.eventRecipient;
                        await that.disableAccount(eventStatusCode, email, that.sql);

                        // Delete message after logging
                        let deleteParams: DeleteMessageRequest = {
                            QueueUrl: queueURL as string,
                            ReceiptHandle: _message.ReceiptHandle as string
                        };
                        that.delete(deleteParams);

                    } catch (e) {
                        LOG.error(e);
                    }
                });
            }

            **// IT WORKS IF listen() is called recursively here
            // Will this cause memory issues?
            // Will it eventually fill the call stack and stack overflow?
            // that.listen();**

        });

    }

Thanks

Gon
  • 11
  • 2
  • the `while` loop is a blocker. – Bekim Bacaj Jan 23 '20 at 03:17
  • See [Why does a while loop block the event loop](https://stackoverflow.com/questions/34824460/why-does-a-while-loop-block-the-event-loop/34825352#34825352). While you are looping in the `while` loop, Javascript cannot process any incoming events. You have to exit the while loop or pause the `while` loop with an `await` to allow Javascript to get back to the event loop and process the next event that is waiting. The key here is that node.js is an event driven system. Things get attention by putting events into the event queue and events only get processed when you return back to the event loop. – jfriend00 Jan 23 '20 at 03:17
  • Other similar references [What is wrong with the while loop never ending loop](https://stackoverflow.com/questions/38676709/javascript-what-is-wrong-with-this-while-loop-never-ending-loop/38676722#38676722) and [Code inside while loop not being executed](https://stackoverflow.com/questions/56916549/code-inside-while-loop-not-being-executed-javascript/56918053#56918053). – jfriend00 Jan 23 '20 at 03:22
  • @jfriend00 Thanks for the useful references. I actually have a second question in this post however, I am not sure if I should create a different post. It's that I actually managed to get it working by using recursion but I am unsure if the first listen() will get "pop off" the call stack or not or it just keeps piling up until stack overflow? Thanks again for useful references. – Gon Jan 23 '20 at 04:05
  • When an asynchronous callback is called, the stack is clean at that point. This is a consequence of how the event system works in node.js. It only gets back to process the next event when the previous event has returned control back to the system (stack is clear). So, you can safely call the function again (recursion-like) from within that asynchronous callback. There is no stack buildup at all. – jfriend00 Jan 23 '20 at 04:29
  • It looks like you may have an issue though because you're doing `async.each()` followed by calling `that.listen()` again. But, you don't wait for the `async.each()` to actually finish before calling `that.listen()` so while you won't get stack build-up, you will get multiple calls to `listen()` all in flight at the same time (which is probably not what you want). So, wait for `async.each()` to be done before going recursive. – jfriend00 Jan 23 '20 at 04:30
  • Thanks, you are right. It's because "that.delete" doesn't return a Promise but executes a callback after. I have tested it with "Messages" array having more than one element and the logs was all over the place which could make it hard for debugging. Awaiting "async.each" to complete does solve the issue. Thanks. – Gon Jan 23 '20 at 08:07

0 Answers0