-1

I'm trying to add a queue mechanism for my micro-service processor that I built, in short, the processor get an id from other server (by REST call), add information from DB and creates a new object with enriched id data. The queue manager should manage an array of id's and enrich data after

x milliseconds OR y amount of id's. (the enrichment will be done as a bulk)

How can I do this? How to manage the queue?

I know that there is setInterval() function which is great but it only solves me the time issue, how can I add another condition to also do the enrichment by array length?

CinCout
  • 9,486
  • 12
  • 49
  • 67
AdirDev
  • 1
  • 1
  • There is a queue Control Flow object in the widely used async library: http://caolan.github.io/async/docs.html#queue - you might want to start there instead of creating your own queue. – Graham Dec 15 '17 at 17:34

1 Answers1

0

To see when your queue hits a certain length, I can think of three strategies for monitoring when the length changes:

  1. Trigger an onChange notification where queue is added to. Make sure your code that adds an item to the queue always does it via one particular function so you don't have lots of random pieces of code modifying the queue. Then, in that one function, right after an item is added to the queue, trigger an onChange event on an EventEmmitter that you create. Then, your queue manager can add a listener for the onChange event and examine the queue each time it is modified.

  2. Override .push() for the queue array. Asssuming you know that items are only added to the queue array with .push(), you would override the .push() method on the queue array with your own method that would call the default push and then you can examine the length of the array to decide if you need to do anything further.

  3. Use an ES6 proxy to actively monitor the length of the array and see anytime it changes. There's an example of how to do that here: https://github.com/gergob/jsProxy/blob/master/04-onchange-object.js and Detecting changes in a Javascript array using the Proxy object

For the time, it sounds like you already know you can use timers such as setInterval() or setTimeout(). It wasn't clear to me how exactly you wanted your time element to work in order to make a more concrete suggestion for how to use those. The sipmlest implementation is probably to just poll the queue with setInterval() every xx ms and process any items currently in the queue. There are likely smarter algorithms that watch when the queue goes from empty to non-empty and starts a setTimeout() from that point in time.

jfriend00
  • 683,504
  • 96
  • 985
  • 979