1

I'm trying to build a backend infrastructure that hits specific API endpoints repetitively on a given time duration, once every second for example. The system would then post (or whatever the appropriate equivalent to a post request would be) the response to my database so that I can work with the data at a later time.

I was thinking Node would work since I am already working with Javascript, as I understand it, it essentially allows JS to run on a server. I want to use MongoDB because it's easier to change the schema vs something like postgreSQL.

With Express you could do something like this when getting all the users (this code is from an api I made so it might not be exact):

router.get('/users', function(req, res, next) {
    const userData = {
        _id: false,
        password: false,
        about: false,
        __v: false
    }
    User.find({}, userData, function(err, user) {
        if(err) {
            res.send('Unable to find user');
            console.log(err);
        } else {
            res.json(user);
        }
    });
});

Obviously the main problem is that this relies on a user going to a route the executes this code, How can I make it so a Node app (or express if possible) can just sit on the server, I can run the file node app.js and then it would constantly run and add the response data to the database (which would be on the same server) without having to worry about going to any pages or otherwise making the code execute?

Please let me know if you have any questions, or would like more clarification.

Garrett
  • 1,576
  • 4
  • 27
  • 51
  • You don't need an express/http-server for this. The two main approaches are to put it on a "cron" that basically executes your script every 5 minutes (`node app.js`) or however often you like, or you can use a "message queue" to process data in the background as it comes in. Google for more info. – mpen Aug 15 '18 at 00:25
  • @mpen I see what you're saying. Just out of curiosity, would using `setInterval` work and then making the executed function be the actual api request with an interval of 1 second and then just calling `node app.js` once on the command line to run it work? – Garrett Aug 15 '18 at 00:28
  • Yeah, it should work as long as you don't have any memory leaks, and your DB connection doesn't go away, and the app doesn't crash. Stuff like that. Re-running the entire process via crontab every minute or so ensures that even if it dies once in awhile it'll just spin back up. It'll also run itself again after a reboot. You can weigh the pros and cons. – mpen Aug 15 '18 at 00:31

1 Answers1

0

Store the code you'd like to have run in a reusable regular function instead coupling it as an anonymous function with the route handler. Then, use a scheduler like node-cron or something similar which calls the function at intervals.

pfcodes
  • 1,055
  • 2
  • 9
  • 15
  • I think I understand. Just out of curiosity, would using `setInterval` work and then making the executed function be the actual api request with an interval of 1 second and then just calling `node app.js` once on the command line to run it work? – Garrett Aug 15 '18 at 00:29
  • @Garrett Yeah probably, but I'd highly recommend against that. It doesn't seem very safe.. Is this code going to live on a cloud platform like AWS/Azure/GCP? If it is, these providers have features where you can write code and have it run at certain intervals. No `setInterval` needed! – pfcodes Aug 15 '18 at 00:32
  • Yeah I'm probably going to put it on Digital Ocean since it's cheap. I'll definitely check if they have that feature. Is there a limit to the interval for a cron job? Could I make it run the program every second or more? – Garrett Aug 15 '18 at 00:45
  • @Garrett There might be a minimum limit (of one minute I think?). How often do you realistically need it to run? – pfcodes Aug 15 '18 at 00:50
  • Ideally I'd like like it to run every second or more, however I might be limited due to server storage limits depending on how much data is actually collected. won't know until I get to testing – Garrett Aug 15 '18 at 21:54