2

I'm using the x-ray library to scrape a specific website for some content and save it in a JSON file.

I'd like to get the status every hour, but Node does not execute my setTimeout.

This is my code:

var db, record, x;
x = require('x-ray')();
db = require('lowdb')('rec.json', {
  storage: require('lowdb/file-async')
});

record = function() {
  console.log('will contact server');
  setTimeout(record, 60 * 60 * 1000);
  x('http://example.com', 'h1')(function(err, content) {
    console.log('server contacted');
    if (err) {
      console.log('error occurred');
      db('status').push({
        time: (new Date).toString(),
        status: err
      });
    } else {
      console.log('status recorded successfully');
      db('status').push({
        time: (new Date).toString(),
        status: content
      });
    }
  });
};
console.log('ready');

record();

It logs the first time without any issues, but after that, it does nothing. It seems that record function is executed only once.

I'm using pm2 to manage the process, pm2 list shows it's running well beyond 60 minutes. Even if Node is unable to execute the callback exactly in the given time, the process was alive (longest, before I restarted it) for around 80 minutes.

There are no errors in the logs and there were no unexpected restarts during my testing.

Maybe this has something to do with the libraries I'm using? I'm out of ideas what could be causing this.

ROAL
  • 1,749
  • 1
  • 14
  • 21
  • I am encountering a similar issue, but for me it seems that after some time the timeouts stop working. Do I understand this right that it only prints 'will contact server' once and never after that? Or did it just never insert anything into the database? – Philiiiiiipp Jul 13 '16 at 09:33
  • It recorded the status to the database, but just once. The whole thing was working as intended, except the timeout. Function executed just once and never after that. However, as I mentioned in a response, I just used cron and slightly modified the code. Might not be applicable in your case, though. – ROAL Jul 13 '16 at 09:39
  • Yeah, but using a cron is not changing the fact that the timeout seems to behave weird. Which version of node where you using? – Philiiiiiipp Jul 13 '16 at 09:50
  • I believe it might have been 5.8.0. – ROAL Jul 13 '16 at 09:54

2 Answers2

0

From the LowDB Docs:

Important

When you modify the database, a Promise is returned.

When you read from the database, the result is immediately returned.

Scheduling the function execution inside the then callback should work:

var db, record, x;
x = require('x-ray')();
db = require('lowdb')('rec.json', {
  storage: require('lowdb/file-async')
});

record = function() {
  console.log('will contact server');
  x('http://example.com', 'h1')(function(err, content) {
    console.log('server contacted');
    if (err) {
      console.log('error occurred');
      db('status').push({
        time: (new Date).toString(),
        status: err
      }).then(function(status) {
        setTimeout(record, 60 * 60 * 1000);
      });
    } else {
      console.log('status recorded successfully');
      db('status').push({
        time: (new Date).toString(),
        status: content
      }).then(function(status) {
        setTimeout(record, 60 * 60 * 1000);
      });
    }
  });
};
console.log('ready');

record();
Community
  • 1
  • 1
Francesco Pezzella
  • 1,755
  • 2
  • 15
  • 18
  • I will try this and accept the answer if it works. Is it important to always have a `then` on Promises? I thought it is not necessary, or am I missing something else? – ROAL Apr 23 '16 at 14:08
  • Still not working. Process up for 8 hours with 0 restarts (I started it and left home). `record` function still executed just once. – ROAL Apr 23 '16 at 22:26
0

Although not an exact solution, I'm posting this as an answer as it solves the issue, and also to close this question.

Ultimately, I solved this using cron.

ROAL
  • 1,749
  • 1
  • 14
  • 21