Running Python Script in Background Infinitely

Question

I am trying to write a python script which runs on another server such that even if I close my server connection on my PC's terminal it keeps on running on that server.When the script is kept alive, it runs infinitely listening to any events on a Website (UI), on event occurrence it then starts up certain dockers appropriately and keeps on listening to PosgreSQL Events.

When I tried to use nohup (to run the script in background) it did run in the background but was unable to listen to any of the events. Has any one worked on something similar before? Please share your thoughts. I am sharing a part of my script.

self.pool = await asyncpg.create_pool(user='alg_user',password='algy',database='alg',host='brain',port=6543)
        async with self.pool.acquire() as conn:
            def enqueue_listener(*args):
                self.queue.put_nowait(args)

            await conn.add_listener('task_created', enqueue_listener)
            print("Added the listener")
            while True:
                print("---- Listening for new job ----")
                conn2, pid, channel, payload = await self.queue.get()
                x = re.sub("[^\w]", " ",  payload).split()
                print(x)
                if x[5] == '1':
                    tsk = 'TASK_ID=%s' % str(x[1])
                    if x[3] == '1':
                        command = "docker run --rm -it -e ALGORITHM_ID=1 -e TASK_ID=%s --network project_default project/docked_prol:1.0" % (str(x[1]))
                        subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)

                    if x[3] == '8':
                        command = "docker run --rm -it -e ALGORITHM_ID=8 -e TASK_ID=%s --network project_default project/docked_pro:1.0" % (str(x[1]))
                        subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)

The script is running absolutely fine when kept on running manually, but just need some advice with implementation methodology.

Answer 1

First of all, I am here after 3 years later.

To run a script infinitely as a background task, you need process manager tools. PM2 is my favorite process manager tool made in nodejs but can run any terminal task because it is a CLI.

1. Basically, you can install NodeJs and npm to reach pm2. (You can visit NodeJs.org to download the installer.)

2. You need to install the pm2 as a global module using npm install -g pm2 on your terminal

3. You can check if it is installed simply by pm2 -v

4. Then you can start your python script on your terminal using pm2 start file_name.py

5. It will create a thread in background to run your script and it will be restart forever.

6. If you were doing something that takes so much time and you dont want to see the task running on the terminal you can just disable restarting by adding the parameter --no-autorestart into the command. (# pm2 start file_name.py --no-autorestart)

7. If you want to see the logs or the state of the task, you can just use pm2 status, pm2 logs and pm2 monit.

8. If you want to stop the task, you can use pm2 stop task_name

9. You can use pm2 reload all or pm2 update to start all the tasks back

10. You can kill the task using pm2 kill

For more information you can visit PM2 Python Documentation

Answer 2

Running something in background via nohup will only work if the process/script runs automatically without providing external inputs, because there is no way to provide manual inputs to a background process. First, try checking if the process is still running in background (ps -fe|grep processname).

If its running, then check the 'nohup.out' file to see where the process is getting stuck. This gets generated in the same directory where you started the process. This will give you some idea what is going on inside the process.