2

Introductory NOTE about the CURRENT STATUS of this question:
In spite of given helpful answers and many helpful comments this question still needs an answer explaining the actual technical reason for the observed behavior instead of providing only high-level explanations and guesses not backed up with detailed knowledge about what is going on under the hood and which scripting language design decisions or system properties result in allowing the child process to finish its job after the script exit.

In other words a good answer should at least give some information related to following detail:

  • is the observed behavior a general rule of behavior or an exception due to special properties of sudo or of Popen? Or in other words: is the reason for the observed behavior caused by the design of Python scripting language or is it caused by the design of the operating system (with no way to shut down a child process preventing it from execution and finishing of still pending tasks at the time of parent shutdown)

See also two other questions which tackle another aspects of this one to get more insight about the context of this question:

How to reliably check from Python script code if just created directory exist?

Why does a later Python code line win the race of becoming executed before similar preceding line?

The code below creates a directory in the root directory of the file system. In order to be able to do it, it needs root rights, so the appropriate code line uses sudo and provides the root password.

Now it turns out that the execution of the line creating the directory waits until the script is finished what causes the code checking if the directory created by the previous lines of code exist report that the directory was not created.

from subprocess import Popen, PIPE
from os.path import isdir
from time import sleep
    
sudoShCmd = Popen(["sudo", "-S", "mkdir", "/AnewDir"], stdin=PIPE) 
sudoShCmd.stdin.write(b'YOUR ROOT PASSWORD HERE')

sleep(5); print()

print(f'{isdir("/AnewDir")=}')

And because I know how I can modify the provided code to preserve the order of executing its lines I am only interested in UNDERSTANDING how it comes that in the case of the code above the last line is executed before the preceding line which seem to be executed at the very end or maybe even after the script shutdown/exit.

So my question is: Why does the Python code creating a directory get executed AFTER the code at the end of the script?

To make even more clear what my question is about here some further clarifying statements:

I am not interested in knowing how to get rid of the observed behavior. I know how to write code that does work as expected. What I am after is to UNDERSTAND what actually happens in case of the code above, so that the lines of code are executed in reversed order.

See my other question on the same subject which current outcome is to ask this one in order to avoid asking multiple question in one question: How to reliably check from Python script code if just created directory exist?.

Here also some thoughts about the explanations given in a comment:

When your script ends, Python begins its cleanup. The subprocess running sudo is still hanging out waiting for input.

Flushing the content of stdin and closing the stdin pipe does not help here. So there must be another reason why sudo does not proceed after getting its password input.

When python starts closing its objects, the stdin closes, the subprocess finally gets control and starts running in parallel. Your script didn't notice that its own subprocess was hanging around waiting for data because the script isn't waiting for the process to finish.

Also with a closed stdin the order of execution will remain switched, so a not closed stdin does not explain the observed behavior.

Claudio
  • 7,474
  • 3
  • 18
  • 48
  • No Python code that is written on an earlier line is executed after other Python code that is written on a later line. Maybe you meant why the directory is created after the Python code is executed that instructs to create the directory. – mkrieger1 Mar 09 '23 at 23:33
  • You didn't `sudoShCmd.stdin.close()` to tell it you're done writing to stdin. Not positive because I haven't tested, but I'd bet that when the script ends, it gets closed automatically, and *then* the external command runs, all while the Python runtime is cleaning up. – Samwise Mar 09 '23 at 23:34
  • The `sudo` command is just sitting there, waiting for you to hit Enter after typing in your password. Adding a newline to the password should fix this. – jasonharper Mar 09 '23 at 23:35
  • Does this answer your question? [Python popen command. Wait until the command is finished](https://stackoverflow.com/questions/2837214/python-popen-command-wait-until-the-command-is-finished) – mkrieger1 Mar 09 '23 at 23:35
  • `sudoShCmd.stdin.write(b'YOUR ROOT PASSWORD HERE')` Are you including a newline at the end of the password? If not, then it sits patiently waiting for you to enter more letters. – John Gordon Mar 09 '23 at 23:36
  • The code DOES create the directory, so the missing Enter or closing the stdin PIPE are not the problem. The problem is that somehow the creation of the directory is postponed AFTER the code that checks for the existence of the directory. Somehow weird ... – Claudio Mar 09 '23 at 23:46
  • I had a newline in the code thinking that this is maybe the reason for the weird behavior, but it didn't have any effect. Sudo HAD received the password and the code created the directory. – Claudio Mar 09 '23 at 23:50
  • @Claudio - you don't wait for the Popen to complete, so the stdin closes and the command executes after your script terminates and python is in its exit processing. – tdelaney Mar 09 '23 at 23:51
  • Try `subprocess.run(["sudo", "-S", "mkdir", "/AnewDir"], input=b'YOUR ROOT PASSWORD HERE\n')`. That should cause stdin to close and the script to wait for you to be done. – tdelaney Mar 09 '23 at 23:52
  • OK ... so I need to close the stdin PIPE ... this should resolve the problem ... I will check it out. – Claudio Mar 09 '23 at 23:53
  • But then you still need to deal with sudo - I think it will reach around stdin to your controlling tty. – tdelaney Mar 09 '23 at 23:55
  • @tdelaney: `run('echo PASSWORD_HERE | sudo -S mkdir /AnewDir 2>/dev/null', shell=True)` is what I am aware of that solves the problem. But ... what I am after is UNDERSTANDING how does it come that code is executed in reversed order. – Claudio Mar 09 '23 at 23:56
  • The code isn't executed in reverse order. You seem to be kinda stuck on that idea. – tdelaney Mar 09 '23 at 23:57
  • The line creating the directory precedes the check for directory existence. The directory existence check reports that the directory is not there and the script exits. So far OK, but ... the directory IS actually CREATED. After the script has exited or in the process of exiting the script? This reverses the order of the executed lines, right? – Claudio Mar 10 '23 at 00:05
  • Not quite. When your script ends, python begins its cleanup. The subprocess running your sudo is hanging out waiting for input. When python starts closing its objects, the stdin closes, the subprocess finally gets control and starts running in parallel. Your script didn't notice that its own subprocess was hanging around waiting for data because the script isn't waiting for the process to finish. – tdelaney Mar 10 '23 at 00:11
  • "The line creating the directory" is not the line creating the directory. It is starting a parallel process that *eventually* will create a directory. Nothing is run out-of-order in the script. – Mark Tolonen Mar 10 '23 at 00:19
  • @MarkTolonen : if I understand you right with the parallel process the order of execution from that point is undetermined as the parallel process acts independently. But why does the script allow the process to continue while it is shutting down itself? – Claudio Mar 10 '23 at 00:35
  • Because the other process is waiting for user input. Only the password was typed, but not Enter. When the main process exists, stdin is flushed. You can add `sudoShCmd.stdin.flush()` but probably need a newline added to the password as well. – Mark Tolonen Mar 10 '23 at 06:21
  • @MarkTolonen : flush() and newline are necessary to prevent the script to hang on the following wait(), but without .wait() the script finishes with the sudo executed after the check for directory existence, so newline and flush() without .wait() don't prevent sudo from being executed while script shutdown. – Claudio Mar 10 '23 at 08:43
  • @tdelaney : see my updated question for evidence that your explanation how it comes that the order of execution is switched does not apply to what actually happens. – Claudio Mar 10 '23 at 09:37
  • Once you properly provide input, then it is a race condition between the two processes which commands execute. The wait guarantees the sudo command completes before moving on in the main script. – Mark Tolonen Mar 10 '23 at 15:20

2 Answers2

1

When you run a subprocess using a PIPE for input, that channel switches from being line oriented to block oriented. Your password was stuck in the pipe and the sudo subprocess is hung waiting for data. You don't wait for the process to complete, so you don't notice the hang. When your process exits, its objects run their delete code. This includes closing the pipe, flushing the password and letting the subprocess run.

You could use subprocess.run instead. It can pass input, close the pipe and wait for process completion. This has the added benefit that you don't need to guess a sleep time for the process to complete.

from subprocess import run
from os.path import isdir
    
result = run(["sudo", "-S", "mkdir", "/AnewDir"],
    input=b'YOUR ROOT PASSWORD HERE',
    capture_output=True)
if result.returncode != 0:
    print("Error", result.stderr) 
print(f'{isdir("/AnewDir")=}')
tdelaney
  • 73,364
  • 6
  • 83
  • 116
  • If the script runs out of lines and is shutting down, why is the sub-process not shut down and allowed to finish its work? It should get the signal to exit not creating the directory, right? – Claudio Mar 10 '23 at 00:25
  • It may complete before the parent finishes exiting, but even if it didn't, I think that the child keeps running and its ownership changes to `init`. If the controlling terminal closes instead, then a SIGHUP would happen, which is a different story. This is just from memory, not not entirely sure. – tdelaney Mar 10 '23 at 00:29
  • It seems that without going the path of the *different story with SIGHUP* the observed behavior can't be properly explained. – Claudio Mar 10 '23 at 09:41
1

Let's summarize the outcome of the other given answer and the great number of comments to the question which made this answer possible (thanks to all who contributed). Hope to have got the idea right, but if not, I would be glad to hear about flaws in this answer in the comments.

Invoking Popen() spawns a parallel process and the script continues its execution by executing next lines of code. Now there are two parallel lines of execution which can act to a given degree independent from each other so the order of finishing the execution of code lines can get reversed, what is the case for the code you provide in your question.

There are two ways to assure that the order of execution of code lines will be preserved:

  • closing the stdin PIPE followed by waiting for the process to exit accomplished by adding following two lines of code below the line with Popen():
sudoShCmd.stdin.close()
sudoShCmd.wait()

If you out-comment only the .close() line you can then experience that the script will 'hang' waiting and not progressing at all. If you out-comment the .wait() line you will still experience the same weird behavior with reversing the order of execution even if the stdin PIPE is explicit closed.

  • using the much more elegant in Python available with construct which takes care about it all as follows:
with Popen(["sudo", "-S", "mkdir", "/AnewDir"], stdin=PIPE) as sudoShCmd: 
    sudoShCmd.stdin.write(b'YOUR_ROOT_PASSWORD\n')

Another simple approach of achieving the same effect will be to use subprocess.run() with the option shell=True echoing the password to the stdin of sudo:

from subprocess import run
run('echo YOUR_ROOT_PASSWORD | sudo -S mkdir /AnewDir 2>/dev/null', shell=True)

The 2>/dev/null part hides the output of sudo which goes to stderr, so that you see only the effect of the print statement as output.

Claudio
  • 7,474
  • 3
  • 18
  • 48