5

Little background of the task- I'm building abandoned cart recovery system for Shopify. After an user makes a checkout, Shopify calls our webhook and the webhook enqueues that request as job with delay of 5mins in a queue A. When worker starts processing that job, it checks if that checkout has been paid yet or not. If it isn't paid yet then it sends cart recovery message to user.

I'm using Node.js, Express.js, Redis & BullMQ to implement the server and queuing system. I have tried to go through basic examples of BUllMQ. On the web, Couldn't find some advanced examples around it to how to use this in production level system.

Right now, I'm stuck with these following questions -

  1. Since, Redis is in-memory database, I must save every incoming job in my MongoDB collection, initially with status of PENDING and listen for completion event to change that status to COMPLETED in the database. Whenever my server restarts, I fetch all the jobs with PENDING status and add them to the queue. This way we can recover our jobs, even if Redis goes down or restarts. My questions are - Does it make sense to do such thing in production level application?, Should I'be saving jobs in MongoDB? & What other caution should I be taking to implement this flow?

  2. Right now, Bull queue(let's name that 'A') is getting initialized in express server. Every time server restarts, queue A also gets initialized. My questions are - Does re-initialization of queue A, removes old queue A on Redis ? What other things I can do around this problem?

I would be very thankful for any help on this.

Aps pk
  • 63
  • 2
  • 6

1 Answers1

2

Let me give you some answers to your questions:

Since, Redis is in-memory database, I must save every incoming job in my MongoDB collection, initially with status of PENDING and listen for completion event to change that status to COMPLETED in the database. Whenever my server restarts, I fetch all the jobs with PENDING status and add them to the queue. This way we can recover our jobs, even if Redis goes down or restarts. My questions are - Does it make sense to do such thing in production level application?, Should I'be saving jobs in MongoDB? & What other caution should I be taking to implement this flow?

Although Redis is an in-memory database you can enable persistence in most cloud providers, or add replication as is standard in AWS Elasticache, which will provide a very reliable system. Of course you should schedule backups for your Redis persisted data as with any database, which all managed Redis providers do. What is important is to keep alarms in case your Redis instance is running out of memory, since that will make your queue stop working. You can enable "removeOnComplete" to avoid the accumulation of jobs that you no longer care about.

Right now, Bull queue(let's name that 'A') is getting initialized in express server. Every time server restarts, queue A also gets initialized. My questions are - Does re-initialization of queue A, removes old queue A on Redis ? What other things I can do around this problem?

No, Bull/BullMQ are designed so that you can instantiate the same Queue as often you want and no data will be lost. In fact that is how you can scale your workers, by instantiating as many as you want for a given queue.

Manuel Astudillo
  • 186
  • 1
  • 12