Using greater than operator with subprocess.Popen

Question

I want to redirect the output of python script to the file using greater than operator. I have below code which is not working properly. Can someone please help me on this?

proc= subprocess.Popen(['python', 'countmapper.py',file],cwd="C:\pythonPrograms\\",stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
countReducer= subprocess.Popen(['python', 'countreducer.py'],cwd="C:\pythonPrograms\\",stdout=subprocess.PIPE,stdin=proc.stdout, stderr=subprocess.STDOUT)
countpostprocesser= subprocess.Popen(['python','countpostprocesser.py','>','output.json'],cwd="C:\pythonPrograms\\",stdout=subprocess.PIPE,stdin=countReducer.stdout,stderr=subprocess.STDOUT)

'file' is name of the log file that I want to process. Last line (starting with countpostprocesser...) is failing.

If your actual scripts are python, a better solution would probably be the multiprocessing module. — deets, Jul 16 '14 at 15:34
Deets, please use the comment fields for comments and not answers. — Alex Reynolds, Jul 16 '14 at 15:58
[plumbum](http://plumbum.readthedocs.org/en/latest/) is perfect for this! — shx2, Jul 16 '14 at 17:07
@AlexReynolds I was under the impression that not answering to the actual question (somehow connecting processes via pipes) but suggesting a completely different approach is actually the place for comments? — deets, Jul 17 '14 at 17:12
It looks like you are trying to emulate `mapper | reducer | processer > output.json` shell command. See [How do I use subprocess.Popen to connect multiple processes by pipes?](http://stackoverflow.com/a/16709666/4279). Take a look at `plumbum` solution there as @shx2 suggested. — jfs, Aug 06 '14 at 21:50

score 3 · Accepted Answer · edited May 23 '17 at 11:58

3

Your call is failing because the redirection operator is being passed to your script as an argument, not being acted on by a shell to redirect your output to a file. See the Popen documentation.

This answer to another SO question shows a good example of opening a file, then redirecting the subprocess' output to the file.

Also, as shx2 mentioned in another answer, passing the shell=True argument to your Popen constructor should accomplish what you're looking for as well. It will cause the process to be opened in it's own shell, allowing the shell program to interpret the arguments you pass. Note an important line in the Popen documentation though: "If shell is True, it is recommended to pass args as a string rather than as a sequence."

edited May 23 '17 at 11:58

Community

1
1

answered Jul 16 '14 at 17:14

skrrgwasme

9,358
11
54
84

I tried this but final 'output.json' file is getting currupted. – Codelearner Jul 16 '14 at 17:34
What does corrupted mean? The output files are broken and can't be opened, or they contain gibberish? I noticed that you're using the*.json extension, so are you encoding your output using the [JSON module](https://docs.python.org/2/library/json.html) in your subprocess then unable to decode the output in some other script? "Corrupted" can mean a lot of things, so you have to be specific. – skrrgwasme Jul 16 '14 at 17:42
Sorry for not detailing out the issue. It worked, I forgot to remove the debugging statements before. It is working fine now. Thanks. – Codelearner Jul 16 '14 at 17:53

score 1 · Answer 2 · answered Jul 16 '14 at 17:10

1

Use the shell=True flag of Popen.

Also, as I mentioned in the comments, your task can be done simply and elegantly using plumbum.

answered Jul 16 '14 at 17:10

shx2

61,779
13
130
153

Using greater than operator with subprocess.Popen

2 Answers2