Python subprocess.Popen() not running command

Question

I'm trying to use subprocess.Popen() to run a command in my script. The code is:

output = Popen(["hrun DAR_MeasLogDump " + log_file_name], stdout=subprocess.PIPE, stderr = subprocess.PIPE, executable="/bin/csh", cwd=cwdir, encoding='utf-8')

When I print the output, it's printing out the created shell output and not the actual command that's in the list. I tried getting rid of executable='/bin/csh', but then Popen wouldn't even run.

I also tried using subprocess.communicate(), but it didn't work either. I would also get the shell output and not the actual command run.

I want to completely avoid using shell=True because of security issues.

EDIT: In many different attempts, "hrun" is not being recoognized. "hrun" is a Pearl script that is being called, DAR_MeasLogDump is the action and log_file_name is the file that the script will call its action on. Is there any sort of set up or configuration that needs to be done in order for "hrun" to be recognized?

Did you try making it a list as it is meant to be: `["hrun DAR_MeasLogDump", log_file_name]` — vahdet, Mar 11 '19 at 19:49
Yes, I tried it, but there is nothing being written to "output". When I run the command on the command line, there is an output which I am hoping to store in "output". — helloworld95, Mar 11 '19 at 19:54

score 1 · Answer 1 · answered Dec 11 '21 at 17:35

I think the problem is that Popen requires a list of every part of the command (command + options), the documentation for Popen inside subprocess has an example for that. So for that line in your script to work, you would need to write it like this:

output = Popen(["/bin/csh", "hrun", "DAR_MeasLogDump", log_file_name], stdout=subprocess.PIPE, stderr = subprocess.PIPE)

I've removed the executable argument, but I guess it could work that way as well.

score 0 · Answer 2 · answered Mar 11 '19 at 19:49

0

Try:

output = Popen(["-c", "hrun DAR_MeasLogDump " +log_file_name], stdout=subprocess.PIPE, stderr = subprocess.PIPE, executable="/bin/csh", cwd=cwdir, encoding='utf-8')

csh is expecting -c "full command here". Without -c I think it just tries to open it as a file.

answered Mar 11 '19 at 19:49

kichik

33,220
7
94
114

Thanks for the reply! I tried doing it this way, and there is nothing being written to "output". When I run the same command on the command line, I see an output, which is what I'm hoping to store to "output". – helloworld95 Mar 11 '19 at 19:53
I got it wrong the first time as I didn't notice `executable="/bin/csh"`. See the updated answer. – kichik Mar 11 '19 at 19:54
I tried it with "-c", but there is still nothing being stored to "output". Is the command not running, or is there something wrong with the arguments being passed into Popen? – helloworld95 Mar 11 '19 at 19:57
Oh `Popen()` doesn't return the output. You still need to call `output.communicate()` after adding `"-c`". You can also try `check_output()` for an easier route. – kichik Mar 11 '19 at 20:00
When using output.communicate(), "DAR_MeasLogDump" is not being recognized (no such file or directory). It's meant to be an action. Surprisingly, "hrun" isn't throwing an error, which was the case for most of the other comments on this thread. – helloworld95 Mar 11 '19 at 21:44

score 0 · Answer 3 · answered Mar 11 '19 at 20:04

Specifying an odd shell and an explicit cwd seems completely out of place here (assuming cwdir is defined to the current directory).

If the first argument to subprocess is a list, no shell is involved.

result = subprocess.run(["hrun", "DAR_MeasLogDump", log_file_name],
    stdout=subprocess.PIPE, stderr = subprocess.PIPE,
    universal_newlines=True, check=True)
output = result.stdout

If you need this to be run under a legacy version of Python, maybe use check_output instead of run.

You generally want to avoid Popen unless you need to do something which the higher-level wrapper functions cannot do.

For (much) more on the nuances of using `subprocess` on Unix, see also https://stackoverflow.com/a/51950538/874188 — tripleee, Mar 11 '19 at 20:07

score 0 · Answer 4 · answered Mar 11 '19 at 20:08

0

You are creating an instance of subprocess.Popen but not executing it.

You should try:

p = Popen(["hrun", "DAR_MeasLogDump ", log_file_name], stdout=subprocess.PIPE, stderr = subprocess.PIPE, cwd=cwdir, encoding='utf-8')

out, err = p.communicate()  # This will get you output

Args should be passed as a sequence if you do not use shell=True, and then using executable should not be required.

Note that if you are not using advanced features from Popen, the doc recommends using subprocess.run:

from subprocess import run

p = run(["hrun", "DAR_MeasLogDump ", log_file_name], capture_output=True, cwd=cwdir, encoding='utf-8')

out, err = p.communicate()  # This will get you output

answered Mar 11 '19 at 20:08

FabienP

3,018
1
20
25

Thanks for the reply! "hrun" is not being found when I try run(). "hrun" is a Pearl script that is being called from another directory that performs the action DAR_MeasLogDump. Is there any kind of set up or configuration that needs to be done in order for run/Popen to recognize hrun? – helloworld95 Mar 11 '19 at 21:46
Then you should call it from its absolute path, or make it available to PATH. And ensure it is executable, with appropriate shebang. – FabienP Mar 12 '19 at 05:41

score 0 · Answer 5 · answered Mar 11 '19 at 20:14

This works with cat example:

import subprocess

log_file_name='-123.txt'

output = subprocess.Popen(['cat', 'DAR_MeasLogDump' + log_file_name], 
                                stdout=subprocess.PIPE, 
                                stderr=subprocess.STDOUT)
stdout, stderr = output.communicate()
print (stdout)
print (stderr)

I think you need only change to your 'hrun' command

score 0 · Answer 6 · edited Apr 29 '22 at 21:10

It seems the same problem that I had at the beginning of a project: you have tried with windows "environment variables". It turns out that when entering the CMD or powershell it does not recognize perl, java, etc. unless you go to the folder where the .exe .py .java, etc. is located and enter the cmd, where the java.exe, python.py, etc. is.

In my ADB project, once I added in my environment variables, I no longer needed to go to the folder where the .exe .py or adb code was located.

Now you can open a CMD and it will execute any command even from your perl , so the interpreter that uses powershell will find and recognize the command.

Python subprocess.Popen() not running command

6 Answers6