0

I am receiving batches of live data from an Apache Kafka message queue every second, and for each batch I need to run a database query within my app. It is crucial for all the queries from the previous batch to be made before the next batch of data is processed. However, the order of execution of the queries per batch do not matter. Due to async, this is much harder.

batchOfRows.on('message', function (data) {
  for (var i = 0; i < batchOfRows.rows.length; i++) {
    query(batchOfRows.rows[i])
      .then(result => console.log(result))
      .catch(error => console.log(error)); 
  }
});

This is giving me unexpected behavior. NOTE: These batches of data are sent every second, forever, so the promises must resolve before the next batch is received dynamically

stark0323
  • 71
  • 1
  • 8
  • 1
    It looks like you need to use a queue. Check: [async.queue](https://caolan.github.io/async/docs.html#queue). And btw, `the promises must resolve before the next batch is received dynamically` you can't guarantee that, you can only make sure that each batch waits before processing another batch. – Marcos Casagrande Jun 22 '18 at 18:08
  • @MarcosCasagrande That will work too. I will check this out and see if it solves my problem. – stark0323 Jun 22 '18 at 18:56

1 Answers1

1

Start a chain of promises.

let promises = Promise.resolve();

As each batch arrives, append a promise to process the batch to that chain...

batchOfRows.on('message', function (data) {
    promises = promises.then(function() {
        let batch = batchOfRows.rows.map(function(row) {
            return query(row);
        });
        return Promise.all(batch).then(function(result) {
            // do something for each completed batch
            console.log('completed a batch')
        });
    });
}
danh
  • 62,181
  • 10
  • 95
  • 136
  • I will try this and let you know of my luck. Thanks! – stark0323 Jun 23 '18 at 12:38
  • Thank you for this response, I found out that I have another issue which I posted about there: https://stackoverflow.com/questions/51001750/promise-closure-within-loop If you don't mind taking a look that would be great. Thanks! – stark0323 Jun 23 '18 at 13:51