Force callbacks to process synchronously?

Question

I’m working with the following code:

for (var x = 1; x < 10000000; x++) {
    count++
    myObj.name = count
    ch.sendToQueue(queueName, new Buffer(JSON.stringify(myObj)));
}

If I comment out the sendToQueue command it finishes without error, if I don’t comment it I get the error: FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory. I believe this is happening because it’s adding my sendToQueue call to a callback queue, these build up and take up memory until there is no memory left.

Is there a way to force the callbacks to run synchronously so they don’t build up?

I’m guessing this may not be the best use case for Node/JavaScript and would love confirmation if that’s the case.

I understand one solution would be to break the loop up into multiple smaller loops, in this particular use case I can’t do that, it needs to be a single long running loop.

We have to know what `sendToQueue` is and probably see the code to know what to suggest. When you run a giant `for` loop, you allow no other node.js code to run during the `for` loop (including even garbage collection) so things may pile up. We need to know what `sendToQueue` is doing and what other processing options for it there might be in order to help. — jfriend00, Dec 08 '16 at 00:49
Probably [amqplib](https://github.com/squaremo/amqp.node)? Fits the signature. Is there anything consuming those messages? Because they have to be stored *somewhere* if they are not, and running out of memory is not surprising at all. — Amadan, Dec 08 '16 at 00:55
Yes amqplib. Also it doesn't really matter what library I call, message queue, database, etc. When I lower the number on the loop I can confirm that the loop finishes prior to executing the code in the library. I'm using RabbitMQ and if I have a consumer or not doesn't make a difference, nothing gets added if I the loop size is is too big. Basically it works or doesn't based on the loop size, it never adds just some items to the queue. — Ken, Dec 08 '16 at 01:06
I don't see any callbacks in your code, and actually it appears to be the *problem* that you're executing all of this synchronously. What you want to do is [break up the processing](http://stackoverflow.com/q/714942/1048572) [in multiple asynchronous chunks](http://stackoverflow.com/q/6864397/1048572). — Bergi, Dec 08 '16 at 01:21

score 0 · Accepted Answer · answered Dec 08 '16 at 01:19

Yeah, you're right, as @jfriend00 says, no other JS code will get executed till you exit the loop if sendToQueue defers. You can chunk the loop, though; something like this should work.

function chunkedLoop(from, to, step, cb) {
  var i = 0;
  for (var i = 0; from < to && i < step; from++, i++) {
    cb(from);
  }
  if (from < to) {
    setTimeout(chunkedLoop.bind(null, from, to, step, cb));
  }
}

chunkedLoop(0, 10, 3, function(i) { console.log(i); });

Every step items, a setTimeout is run to defer the rest of the loop, allowing any other processing to happen.

Follow up question, if I make the loop big enough I need to make the timeout more seconds to allow what is queue'd to finish. Is there a way to query how many things are in queue so I can wait for it to drain before kicking off the next run? — Ken, Dec 19 '16 at 14:46

Force callbacks to process synchronously?

1 Answers1