nodejs setTimout loop stopped after many weeks of iterations

Question

Solved : It's a node bug. Happens after ~25 days (2^32 milliseconds), See answer for details.

Is there a maximum number of iterations of this cycle?

function do_every_x_seconds() {
    //Do some basic things to get x
    setTimeout( do_every_x_seconds, 1000 * x );                                                                
};

As I understand, this is considered a "best practice" way of getting things to run periodically, so I very much doubt it.

I'm running an express server on Ubuntu with a number of timeout loops . One loop that runs every second and basically prints the timestamp. One that calls an external http request every 5 seconds and one that runs every X 30 to 300 seconds.

It all seemes to work well enough. However after 25 days without any usage, and several million iterations later, the node instance is still up, but all three of the setTimout loops have stopped. No error messages are reported at all.

Even stranger is that the Express server is still up, and I can load http sites which prints to the same console as where the periodic timestamp was being printed.

I'm not sure if its related, but I also run nodejs with the --expose-gc flag and perform periodic garbage collection and to monitor that memory is in acceptable ranges.

It is a development server, so I have left the instance up in case there is some advice on what I can do to look further into the issue.

Could it be that somehow the event-loop dropped all it's timers?

Moorglade · Accepted Answer · 2018-08-16T05:58:44.447

3

I have a similar problem with setInterval().

I think it may be caused by the following bug in Node.js, which seem to have been fixed recently: setInterval callback function unexpected halt #22149

Update: it seems the fix has been released in Node.js 10.9.0.

edited Aug 16 '18 at 05:58

answered Aug 14 '18 at 07:21

Moorglade

316
2
8

Yep. That looks like exactly the issue. Nice to be on the front line. Thanks for the reference. – SpiRail Aug 14 '18 at 10:20

score 2 · Answer 2 · answered Aug 13 '18 at 16:37

2

I think the problem is that you are relying on setTimeout to be active over days. setTimeout is great for periodic running of functions, but I don't think you should trust it over extended time periods. Consider this question: can setInterval drift over time? and one of its linked issues: setInterval interval includes duration of callback #7346.

If you need to have things happen intermittently at particular times, a better way to attack this would be to schedule cron tasks that perform the tasks instead. They are more resilient and failures are recorded at a system level in the journal rather than from within the node process.

A good related answer/question is Node.js setTimeout for 24 hours - any caveats? which mentions using the npm package cron to do task scheduling.

answered Aug 13 '18 at 16:37

zero298

25,467
10
75
100

1

_"`setTimeout` is great for periodic running of functions"_. That's exactly what OP is using it for: one that runs a function after 5 seconds, and one that runs a function after 30-300 seconds. The callback functions then schedule a _new_ timeout. – robertklep Aug 13 '18 at 16:46
This is a good answer in terms of how it should be done. I'm also looking for why this happened to be sure its not something else in the system. In terms of drift, it depends on the requirements of your application. I solve this by checking the processing times of functions, as well as checking the date and adjusting accordingly. This keeps my time critical functions within 10ms over its entire runtime (at least as far as I am aware). – SpiRail Aug 14 '18 at 09:54

nodejs setTimout loop stopped after many weeks of iterations

2 Answers2

Linked