17

My goal is simple: kick off rsync and DO NOT WAIT.

Python 2.7.9 on Debian

Sample code:

rsync_cmd = "/usr/bin/rsync -a -e 'ssh -i /home/myuser/.ssh/id_rsa' {0}@{1}:'{2}' {3}".format(remote_user, remote_server, file1, file1)
rsync_cmd2 = "/usr/bin/rsync -a -e 'ssh -i /home/myuser/.ssh/id_rsa' {0}@{1}:'{2}' {3} &".format(remote_user, remote_server, file1, file1)
rsync_path = "/usr/bin/rsync"
rsync_args = shlex.split("-a -e 'ssh -i /home/mysuser/.ssh/id_rsa' {0}@{1}:'{2}' {3}".format(remote_user, remote_server, file1, file1))
#subprocess.call(rsync_cmd, shell=True)     # This isn't supposed to work but I tried it
#subprocess.Popen(rsync_cmd, shell=True)    # This is supposed to be the solution but not for me
#subprocess.Popen(rsync_cmd2, shell=True)   # Adding my own shell "&" to background it, still fails
#subprocess.Popen(rsync_cmd, shell=True, stdin=None, stdout=None, stderr=None, close_fds=True)  # This doesn't work
#subprocess.Popen(shlex.split(rsync_cmd))   # This doesn't work
#os.execv(rsync_path, rsync_args)           # This doesn't work
#os.spawnv(os.P_NOWAIT, rsync_path, rsync_args) # This doesn't work
#os.system(rsync_cmd2)                      # This doesn't work
print "DONE"

(I've commented out the execution commands only because I'm actually keeping all of my trials in my code so that I know what I've done and what I haven't done. Obviously, I would run the script with the right line uncommented.)

What happens is this...I can watch the transfer on the server and when it's finished, then I get a "DONE" printed to the screen.

What I'd like to have happen is a "DONE" printed immediately after issuing the rsync command and for the transfer to start.

Seems very straight-forward. I've followed details outlined in other posts, like this one and this one, but something is preventing it from working for me.

Thanks ahead of time.

(I have tried everything I can find in StackExchange and don't feel like this is a duplicate because I still can't get it to work. Something isn't right in my setup and need help.)

martineau
  • 119,623
  • 25
  • 170
  • 301
harperville
  • 6,921
  • 8
  • 28
  • 36
  • 1
    Seems to the question is answered [here](http://stackoverflow.com/a/2251026/3070670) – Viach Kakovskyi May 05 '16 at 18:50
  • @viach's pointer is your best bet. – WreckeR May 05 '16 at 18:54
  • [This one](http://stackoverflow.com/questions/3516007/run-process-and-dont-wait#comment19551411_6700359) says "creationflags" is Windows only. The link you supplied talks about Win32, too. – harperville May 05 '16 at 18:54
  • 1
    Have you tried simplifying your code to see where the problem really is? `subprocess.call("sleep 10".split())` has an obvious delay in the code whereas `subprocess.Popen("sleep 10".split())` proceeds to the next line immediately. Maybe your problem is somewhere else and hard to see? Does your code work (in terms of timings) if you changed your rsync call to a simple sleep call? – Chris May 05 '16 at 18:59
  • can you try using `Process()` instead `subprocess` ? just don't wait for `join()` when using `Process()` – Asav Patel May 05 '16 at 19:14
  • 1
    unrelated: it is *useless* to mention *"NOTHING that works for me"* unless you show the exact code example that fails, describe what did you expect it to do and what happens instead step by step. – jfs May 05 '16 at 20:25
  • What jfs said -- if you want to assert that the things that look like duplicates aren't, link to them from the question, showing each thing you tried and how it failed in enough detail to let us identify implementation issues in same. Very often something *truly is* a duplicate, but an answer's advice is being interpreted or applied incorrectly, or an answer wasn't written to cover a corner case and can be fixed or improved there rather than having a whole new overlapping question in the knowledgebase alongside. – Charles Duffy May 04 '18 at 16:48
  • Possible duplicate of [How to start a background process in Python?](https://stackoverflow.com/questions/1196074/how-to-start-a-background-process-in-python) – Ciro Santilli OurBigBook.com Dec 12 '18 at 14:31
  • how is something like this not a standard example in the documentation? – Charlie Parker Feb 24 '19 at 19:02
  • @CharlieParker is the reference to `execvp()` in the [docs](https://docs.python.org/3.2/library/subprocess.html#popen-constructor) supposed to tip one off that it "runs as its own process in the background"? – harperville Feb 26 '19 at 14:07

4 Answers4

19

Here is verified example for Python REPL:

>>> import subprocess
>>> import sys
>>> p = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(100)'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT); print('finished')
finished

How to verify that via another terminal window:

$ ps aux | grep python

Output: 

user           32820   0.0  0.0  2447684   3972 s003  S+   10:11PM   0:00.01 /Users/user/venv/bin/python -c import time; time.sleep(100)
Viach Kakovskyi
  • 1,487
  • 1
  • 14
  • 21
  • 1
    This works for me in the interpreter as well, but not my script. I'll have to start tearing it down to nothing and build it back up. I'll also try `Process()` as recommended by @asav above. – harperville May 05 '16 at 19:24
  • 1
    I printed a message to the log after calling Popen as your example shows and I saw my message in the log. The issue is elsewhere that I have yet to find but as usual, your suggestion helped me determine Popen was not the issue. Thanks for the help! I am using `p = subprocess(rsync_dict, close_fds=True)` is it works as advertised. – harperville May 05 '16 at 19:30
  • 2
    @harperville I feel like the reason of the behavior is in your command, which probably waits for some input from `stdin`. – Viach Kakovskyi May 05 '16 at 19:32
  • 1
    use stdout=DEVNULL instead of stdout=PIPE otherwise the child process may stall when the OS pipe buffer fills up. – jfs Jun 12 '18 at 18:45
  • the answer by jfs implies that just creating an instance of `Popen` makes it detached by itself or at least does not make your python script wait. If that is the case why does your answer seem much more complicated than just that? Also, in the original question you posted a link that references OS depedencies. Is your answer OS agnostic? – Charlie Parker Feb 24 '19 at 19:16
  • @CharlieParker I don't see the word "detach" in my answer. It is trivial to check that `Popen()` does not wait for the child process to exit, you don't have to take my word for it, you can read the Popen() docs (what it promises), test it (how it actually behaves in your environment). It depends, what additional steps might be required (you don't need to create a full daemon in every single case). For more details, you could follow the links in my answer. – jfs Jun 11 '19 at 15:47
13

Popen() starts a child process—it does not wait for it to exit. You have to call .wait() method explicitly if you want to wait for the child process. In that sense, all subprocesses are background processes.

On the other hand, the child process may inherit various properties/resources from the parent such as open file descriptors, the process group, its control terminal, some signal configuration, etc—it may lead to preventing ancestors processes to exit e.g., Python subprocess .check_call vs .check_output or the child may die prematurely on Ctrl-C (SIGINT signal is sent to the foreground process group) or if the terminal session is closed (SIGHUP).

To disassociate the child process completely, you should make it a daemon. Sometimes something in between could be enough e.g., it is enough to redirect the inherited stdout in a grandchild so that .communicate() in the parent would return when its immediate child exits.

Community
  • 1
  • 1
jfs
  • 399,953
  • 195
  • 994
  • 1,670
1

I encountered a similar issue while working with qnx devices and wanted a sub-process that runs independently of the main process and even runs after the main process terminates. Here's the solution I found that actually works 'creationflags=subprocess.DETACHED_PROCESS':

import subprocess
import time

pid = subprocess.Popen(["python", "path_to_script\turn_ecu_on.py"], creationflags=subprocess.DETACHED_PROCESS)

time.sleep(15)
print("Done")

Link to the doc: https://docs.python.org/3/library/subprocess.html#subprocess.Popen

Paul Resiga
  • 31
  • 2
  • Any idea why this flag doesn't work on ubuntu python3.7 ? – Shakeel Sep 17 '20 at 09:00
  • Because according to the code, the subprocess module only defines this flag for win32 platform and python versions 3.7 and above. It's a windows constant (https://docs.python.org/3/library/subprocess.html#windows-constants) – dhanshreeA Jan 15 '21 at 08:16
0

In Ubuntu the following commands keep working even if python app exits.

url = "https://www.youtube.com/watch?v=t3kcqTE6x4A"  
cmd = f"mpv '{url}' && zenity --info --text 'you have watched {url}' &"
os.system(cmd)
podcast
  • 131
  • 7