0

iam programming a prototype application with the following components:

  • webinterface for admins
  • doing cron jobs(statistic generation, ..)
  • interact with other webservices over http

I started programming with nodejs(typescript) and i got the connection to the other services. Now i got a problem with cron-jobs in nodejs. Iam using node-cron for executing the cronjob.

Inside one job i need to obtain the status of much pc's and make a summary of it. If i would do this, this would block the main thread. So i think I need to this in a separate thread.

How can i do this in nodejs? Should i use webworker-threads? Am I on the proper way? Should i better use Java(Grails/Spring) for this? I really like the simplicity of nodejs (for http clients, ..)

Hope someone can give me hope that iam on the proper way.

wot-stefan
  • 91
  • 7
  • It sounds like you are concerned that the webservice calls are going to block Node's main thread... They won't, see: http://stackoverflow.com/a/14797359/311181. – chardy Jul 08 '16 at 15:08
  • @chardy : not entirely true. In Node.js everything runs in parallel, except your code. What this means is that all I/O code that you write in Node.js is non-blocking, while (conversely) all non-I/O code that you write in Node.js is blocking. – Rudy Jul 11 '16 at 11:36
  • Well I decided to use https://github.com/rschmukler/agenda because it better fits my needs. Together with agenda i will use the clustering option of nodejs (described by @Rudy). The problem with statistic statistic generating is that it's difficult to make it non blocking. For example: if you iterate over some data with a for then it blocks. – wot-stefan Jul 11 '16 at 13:45
  • @rudy : I agree with your statement about I/O vs non I/O. I was explicitly talking about the web service calls. When you call to a web service, you are mostly waiting on network I/O. The actual amount of code execution that blocks the thread is nominal. – chardy Jul 11 '16 at 21:56
  • @chardy I agree that webservice is an I/O operation, however it depends on what you do later on after you get your webservice result. If it takes time + no async functions at all, then you start blocking. – Rudy Jul 12 '16 at 03:34

1 Answers1

0

I will just use Node Cluster. Using cluster, a master can create a multiple workers, which means your cron wont block incoming request. Just make sure that only 1 worker doing the Cron.

I have never working with node-cron before, but I have experience with the SyncedCron. But should be the same.

For the http client there are a lot libraries doing this, you can check Request or httpclient.

Your code should look something like this :

var cluster = require('cluster');
var http = require('http');
var numWorkers = require('os').cpus().length-1; // just give 1 cpu for OS to use, or maybe 2

if (cluster.isMaster) {
   console.log('Master cluster setting up ' + numWorkers + ' workers...');
   var cronPID=null; 

   for(var i = 0; i < numWorkers; i++) {
     var worker=cluster.fork();
     if(i==0){
      //instructing the first worker to assume role of SyncedCron.
      worker.send('you do it!');
      cronPID=worker.process.pid;
      console.log("worker "+cronPID+" choosed as Cron worker!");
     }
   }
   cluster.on('online', function(worker) {
       console.log('Worker ' + worker.process.pid + ' is online');
   });
   cluster.on('exit', function(worker, code, signal) {
       // have to revive the worker
       console.log('Worker ' + worker.process.pid + ' died with code: ' + code + ', and signal: ' + signal);
       console.log('Starting a new worker');
       var newWorker=cluster.fork();
       if(cronPID==worker.process.pid)
       {   // need to re-elect a new cron worker! 
           newWorker.send('you do it!');
           cronPID=newWorker.process.pid;
           console.log("worker "+cronPID+" choosed as Cron worker!"); 
       }  
   });
}else
{  // worker sides
   process.on('message', (msg) => {
      // validate your message that you get
      // if validated, create a cron job here
   });
   // create your express below, I assume you use express instead of node's http library
   var express = require('express');
   var app = express();
   app.post...........

} 

Note :

  1. To revive the master, use something like "forever"
  2. Your server should have multiple core, at least 4 but I recommend more (8 maybe?).
Rudy
  • 7,008
  • 12
  • 50
  • 85
  • Well I decided to use https://github.com/rschmukler/agenda because it better fits my needs. Together with agenda i will use the clustering option of nodejs (described by @Rudy). The problem with statistic statistic generating is that it's difficult to make it non blocking. For example: if you iterate over some data with a for then it blocks. – wot-stefan Jul 11 '16 at 13:46