0

I have built a simple application (loop) with node and phantom.js that scrapes the web for some data and emails me periodically with new results.

Currently I run the command in my local terminal and as long as the window is open it will continue to run. My question is how should I go about implementing an interface (web app most likely) so that I can type in my parameters and have it run even when I leave the page?

I've heard talk of a package called 'forever' but I'm unsure of how it works if maybe someone could explain that to me.

I have my own shared hosting but conceptually I'm confused about how I could implement this. I can push my code to heroku but from there should I maybe create a simple dashboard for me to start and stop my web scraper loop? Heroku needs to spin up after like an hour of inactivity so that might not work.

My app currently just picks up the parameters from the command line so I'd obviously have to instead grab the data from input fields or something.

High level concept of what needs to be done would be great help.

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Taylor A. Leach
  • 2,115
  • 4
  • 25
  • 43
  • https://stackoverflow.com/questions/19282779/send-data-to-node-js-server-from-html-form Here you have some info how to send data from an HTML form to your server. As for not closing the app when your terminal is not working, you can use `screen` on ubuntu. – ClaudiusDan Jul 10 '18 at 06:05

3 Answers3

2

I think PM2 is the answer :)

You can manage your node app in your system for instance you can start a node project: 

pm2 start app.js

and for listing your current process

pm2 list

and so on ...

servatj
  • 196
  • 2
  • 10
  • How would this help me hook my application up to the front end with inputs for me to turn on and off my scraper? – Taylor A. Leach Jul 10 '18 at 12:32
  • You can always create an express app, expose some endpoints and create a front end app with a form to send parameters via those endpoints for example. I think you have to stop first and think about what you really want to do. There are too many possibilities ... Good Luck! – servatj Jul 10 '18 at 13:10
  • I am comfortable doing all of that but what confuses me and the reason for my post was I am unsure how to keep my server function alive after I have closed the front end web page... – Taylor A. Leach Jul 10 '18 at 13:27
1

I have never used Heroku, but you can create an express server and publicly expose the application using Ngrok.

You can create an html page and access the app via Ngrok that way.

Here is a link: https://ngrok.com/

Quick overview:
1) Create and run a local express server
2) run ngrok tunnel service (ngrok http server port)
3) ngrok will then respond with a url, kinda looks like a hash code, which you can use to access your app from anywhere

I am not sure how long ngrok public server will stay alive for though, in my experience it has stayed on as long as I have the process running.

Other options are straight up hosting the application on a web service such as AWS, and exposing endpoints - but if you are not familiar with that there is some learning involved.

NutellaAddict
  • 574
  • 6
  • 24
  • I've used ngrok before but for testing websites on my other devices to tunnel through to my localhost. I don't think you understood what I was trying to accomplish and am not really sure how ngrok would help in this situation. I'm looking for a way to have my application live and accessible not just on my local machine. – Taylor A. Leach Jul 10 '18 at 12:34
  • ngrok exposes it outside your local machine. – NutellaAddict Jul 12 '18 at 06:57
0

You could use Glitch for hosting. It's free, isn't going anywhere (so you don't need to worry about it only being a temporary solution), and works great for Node applications, although there are some technical restrictions (you'd need to set something up to automatically ping your application, just to keep it running).

Just set up a webpage using something like express with a couple of inputs and start/stop buttons. Those would sent requests to the server (GET requests would probably be the easiest to work with) which could then run functions with your inputs as the arguments when you click start, and when you click stop it could either cancel the function somehow or just kill the node process.

That way you should able to run your app for a (theoretically) infinite amount of time, even if your computer is off. Things like PM2 and Forever will keep your app running while the window is closed, but unless there's some sort of magic involved they won't do much if your computer is turned off as far as I know.

dragonwocky
  • 61
  • 5
  • I feel like your explanation is the closest thing to what I'm aiming for. What I don't fully understand is how I would be able to keep the process running after I have closed my express application. Using Forever or PM2, the host would never be turned off so wouldn't that work? Again, I don't just want my app to run locally I ideally want to host it on my shared hosting or Heroku or something. – Taylor A. Leach Jul 10 '18 at 12:37
  • @TaylorA.Leach basically, to summarise my deleted comments - Glitch works slightly differently so it can be kept it free (but still work well), and if you set up something to auto-ping your project every few minutes it should stay online without Forever or PM2. If you plan to run Node on something other than Glitch, you probably would need one of them. Express runs on the Node server and simply responds to requests, though. It doesn't require the frontend to be kept open at all. Don't worry about your application stopping when you close your browser. – dragonwocky Jul 10 '18 at 14:29
  • Thanks for the information. I will create a simple express app and put it on Heroku because I am familar with that setup. If I ping the app at intervals it will prevent the dyno from sleeping. I think my confusion seeded from me not fully understanding that calling one of my node functions in the server file will run continuously until I stop it manually. I guess by using a flag to stop the continuous loop on a button press or something. – Taylor A. Leach Jul 10 '18 at 15:30