0

I have a long-running python script that I want to run on my AWS instance (with Ubuntu), and leave it running after I close the SSH connection so that I can come the next day and look at the results.

I usually do python long_running_script.py & which sometimes works, but sometimes the process ends abruptly. The script writes to a log file and there are no errors reported in the log file, it just stops logging (I have also redirected stderr to the log file so any error should show up there, and it does if it stops due to a bug in the code or keyboard interrupt).

The script has multiprocessing code, but it only uses 4 out of the 16 cores available and ~40% memory, so I don't think it stops due to excessive resource usage. I use the in-built terminal of VSCode to execute the above-mentioned command and run my script.

The instance itself does not shut down and is always running. The EC2 management console monitoring does not show any crashes and the usage simply drops to 0. The AWS account is not my personal account and is provided by my university through cloudbank if that makes a difference.

I want to know what is the reason for this behavior, especially the fact that sometimes 3-4 day long scripts also finish successfully and sometimes the process terminates very soon after I close my SSH connection. Also, what is the recommended way of running long scripts like this on a remote server?

k-c
  • 67
  • 5
  • 3
    Try `nohup python long_running_script.py &` – Philippe Jun 19 '23 at 22:27
  • The suggested nohup, screen, and tmux options might be OK for a basic developer environment but they aren't good options for running an actual server app on Linux. Look at options such as pm2, init, or systemd. – jarmod Jun 20 '23 at 21:03
  • @Philippe would it make a difference if I used sudo? `sudo nohup python long_running_script.py &` – k-c Jun 21 '23 at 18:26
  • Try `nohup sudo python long_running_script.py &` You need to setup password-less sudo. – Philippe Jun 21 '23 at 18:42

1 Answers1

3

There are additional tools such as screen or tmux that can help you. Here is some documentation for them:

Vladimir Mukhin
  • 46
  • 1