Python subprocess.call not waiting for process to finish blender

Question

I have a python script in blender where it has

subprocess.call(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)

followed by many other code which depends on this shell script to finish. What happens is that it doesn't wait for it to finish, I don't know why? I even tried using Popen instead of call as shown:

p1 = subprocess.Popen(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
p1.wait()

and I tried using commuincate but it still didn't work:

p1 = subprocess.Popen(os.path.abspath('D:/Test/run-my-script.sh'),shell=True).communicate()

this shell script works great on MacOS (after changing paths) and waits when using subprocess.call(['sh', '/userA/Test/run-my-script.sh'])

but on Windows this is what happens, I run the below python script in Blender then once it gets to the subprocess line Git bash is opened and runs the shell script while blender doesn't wait for it to finish it just prints Hello in its console without waiting for the Git Bash to finish. Any help?

import bpy
import subprocess
subprocess.call(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
print('Hello')

What happens on windows when you run `D:/Test/run-my-script.sh` from the command line? Does it return right away? — AShelly, Jun 07 '17 at 19:03

joebeeson · Answer 1 · 2017-06-12T15:09:27.337

You can use subprocess.call to do exactly that.

subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None)

Run the command described by args. Wait for command to complete, then return the returncode attribute.

Edit: I think I have a hunch on what's going on. The command works on your Mac because Macs, I believe, support Bash out of the box (at least something functionally equivalent) while on Windows it sees your attempt to run a ".sh" file and instead fires up Git Bash which I presume performs a couple forks when starting.

Because of this Python thinks that your script is done, the PID is gone.

If I were you I would do this:

Generate a unique, non-existing, absolute path in your "launching" script using the tempfile module.
When launching the script, pass the path you just made as an argument.
When the script starts, have it create a file at the path. When done, delete the file.
The launching script should watch for the creation and deletion of that file to indicate the status of the script.

Hopefully that makes sense.

There are cases when this does not work. In my case for example the process is a program that creates a file and then it access the file to modify its content for a period of time. subprocess.call sees the process as completed, when the file is created while the program is still modifying the content of the file and the process is still progressing. Any workaround available? — SeF, Jun 08 '17 at 08:23
@SeF Then you're doing something unexpected -- the documentation very explicitly states how the function works. Perhaps you're forking or executing to another process? — joebeeson, Jun 08 '17 at 13:44

score 8 · Answer 2 · edited Jun 14 '17 at 22:10

8

You can use Popen.communicate API.

p1 = subprocess.Popen(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
sStdout, sStdErr = p1.communicate()

The command

Popen.communicate(input=None, timeout=None)

Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for the process to terminate.

edited Jun 14 '17 at 22:10

SeF

3,864
2
28
41

answered Jun 14 '17 at 05:59

Pankaj

181
1
3

Thanks to all who participated to the bounty, this answer is proposing an alternative to the wait command that does not appear to work all the time. – SeF Jun 14 '17 at 21:10

score 2 · Answer 3 · answered Apr 11 '17 at 23:56

2

subprocess.run will by default wait for the process to finish.

answered Apr 11 '17 at 23:56

Grr

15,553
7
65
85

Your question gave me the inpression you were using `subprocess.Popen` not `subprocess.run` does it not wait when using run as well? – Grr Apr 12 '17 at 00:28
I used subprocess.run and it didn't wait as well – Tak Apr 12 '17 at 00:29
what happens if you use one of the `os.spawn` methods with `os.P_WAIT` as the first arg? – Grr Apr 12 '17 at 01:57
what do you mean by "as the first arg"? – Tak Apr 12 '17 at 02:02
the first argument. before the argument for the command you are trying to call. It's the old way of starting a process and telling python to wait for it to finish. So for example `os.spawn(os.P_WAIT, "run_my_script.sh")` – Grr Apr 12 '17 at 02:37
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/141484/discussion-between-tak-and-grr). – Tak Apr 12 '17 at 02:40
there is no `os.spawn` there is `os.spawnl,spawnle,spawnv,spawnve` – Tak Apr 12 '17 at 02:45

score 2 · Answer 4 · answered Jun 12 '17 at 06:55

2

Use subprocess.Popen and Popen.wait:

process = subprocess.Popen(['D:/Test/run-my-script.sh'],shell=True, executable="/bin/bash")
process.wait()

You could also use check_call() instead of Popen.

answered Jun 12 '17 at 06:55

Fran Raga

624
7
20

score 1 · Answer 5 · answered Jun 09 '17 at 16:14

1

You can use os.system, like this:

import bpy
import os
os.system("sh "+os.path.abspath('D:/Test/run-my-script.sh'))
print('Hello')

answered Jun 09 '17 at 16:14

A. STEFANI

6,707
1
23
48

`os.system` is nice in some cases, but not always. He may only have an error while calling his binary, so suggesting to use os.system is not really relevant. – David Peicho Jul 20 '18 at 22:32

score 1 · Answer 6 · answered Jun 14 '17 at 21:08

There are apparently cases when the run command fails. This is my workaround:

def check_has_finished(pfi, interval=1, timeout=100):
    if os.path.exists(pfi):
        if pfi.endswith('.nii.gz'):
            mustend = time.time() + timeout
            while time.time() < mustend:
                try:
                    # Command is an ad hoc one to check if the process  has finished.
                    subprocess.check_output('command {}'.format(pfi), shell=True)
                except subprocess.CalledProcessError:
                    print "Caught CalledProcessError"
                else:
                    return True
                time.sleep(interval)
            msg = 'command {0} not working after {1} tests. \n'.format(pfi, timeout)
            raise IOError(msg)
        else:
            return True
    else:
        msg = '{} does not exist!'.format(pfi)
        raise IOError(msg)

score 0 · Answer 7 · answered Aug 29 '18 at 09:29

A wild try, but are you running the shell as Admin while Blender as regular user or vice versa?

Long story short (very short), Windows UAC is a sort of isolated environment between admin and regular user, so random quirks like this can happen. Unfortunately I can't remember the source of this, the closest I found is this.

My problem was the exact opposite of yours, the wait() got stuck in a infinite loop because my python REPL was fired from an admin shell and wasn't able to read the state of the regular user subprocess. Reverting to normal user shell got it fixed. It's not the first time I'm bit from this UAC snafu.

Python subprocess.call not waiting for process to finish blender

7 Answers7