1

Edit: Oops! I accidentally posted code that didn't match my question. I started writing this post before I was done experimenting, so I posted code from an intermediate point in my testing process. I've amended this question to reflect what I really meant to post by changing the code slightly.

The following Python code works as expected with one problem: unless I If verbose == False, the file produced by the Perl script is not created. Why do I have to call output.stdout.read() for the underlying Perl script to successfully create a file?

cmdStringList = ["perl","script.pl","arg1",...]

output = subprocess.Popen(cmdStringList,stdout=subprocess.PIPE)

if verbose:
    print output.stdout.read()

I didn't even realize anything was wrong until I tried to execute my Python script with verbose=False in a production environment. I did some google-fu to try to understand the behavior of Popen and subprocess, but I haven't come up with a reason for this behavior. Any help would be greatly appreciated.

Anthony
  • 1,760
  • 1
  • 23
  • 43
  • Your question is inconsistent. You said not calling `output.stdout.read()` causes it to fail, and you also said it started to fail when verbose was set to false. Which one is it? Did you post the wrong code? – ikegami Feb 07 '13 at 00:26
  • You're right - I messed up my question. I changed it to have the code that I initially meant to post. – Anthony Feb 07 '13 at 03:53

2 Answers2

1

You actually have the following code, right?

if verbose:
    outputRead = output.stdout.read()
    print outputRead

It's quite likely that the problem is that the child is blocked trying to write to STDOUT. Until you make some space in the pipe by reading from it, the child won't be able to finish writing to STDOUT and move on to where it creates the file to which you refer.

If you want to prevent the child from blocking without reading, redirect its stdout to nul (Windows) or /dev/null (elsewhere). This should work everywhere but Windows:

cmdStringList = ["sh","-c","\"$0\"" \"$@\" >/dev/null","perl","script.pl","arg1",...]

(Pardon any syntax errors. I don't know Python at all.)

ikegami
  • 367,544
  • 15
  • 269
  • 518
  • Yes, you have it right. My initial code was accidentally wrong because of a timing error on my part. I didn't transfer it across a variable in the code I meant to post, though. I directly printed output.stdout.read(), which is now shown in my question. – Anthony Feb 07 '13 at 03:55
  • This makes sense I think. I'm not sure how to confirm that it's the problem, though. – Anthony Feb 07 '13 at 16:06
  • Does the Perl program print stuff out before creating the file? If so, there you go. – ikegami Feb 07 '13 at 16:22
1

You should keep reading a stream, otherwise the buffer may become full, then your perl process may get blocked. It has nothing to do with a specific language but the underyling operating system.

I encountered a similar problem when I use Java.

Community
  • 1
  • 1
StarPinkER
  • 14,081
  • 7
  • 55
  • 81
  • How do I continuously read the pipe if I don't want to block program execution? Should I use the mutliprocessing module? – Anthony Feb 07 '13 at 16:04
  • That would be a waste. Just redirect the output to nul or /dev/null, like I mentioned. – ikegami Feb 07 '13 at 16:23