0

How do I force subprocess.Popen/os.Popen to write large output directly to file, without holding anything in the buffer ? I tried this:

os.popen(cmd, bufsize=0)

it didn`t help. Any solution of reading all the output (again, large output) and saving it to file/list would help.

jfs
  • 399,953
  • 195
  • 994
  • 1,670
user2234234
  • 349
  • 1
  • 7
  • 18

2 Answers2

2

This is how to redirect stdout to a file:

import sys
sys.stdout = open('your_file', 'w')

Then everything that would have outputted in the shell is dumped in that file.

anon582847382
  • 19,907
  • 5
  • 54
  • 57
  • It didn`t work. the process is printing ~500 lines to the file and holds the rest in buffer. when I kill the script, it write the rest to the file. – user2234234 Mar 25 '14 at 16:58
  • You can reopen the `stdout` file descriptor with write mode, and 0 as the buffer size (so unbuffered). `sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)`. – anon582847382 Mar 25 '14 at 17:02
  • replacing `sys.stdout` does not redirect the output from a subprocess to the file. You need either use `stdout` parameter of a subprocess or [redirect stdout at file descriptor level, see `stdout_redirected()`](http://stackoverflow.com/a/22434262/4279) – jfs Mar 25 '14 at 17:10
2

You could use stdout parameter to redirect output from a subprocess to a file:

import shlex
from subprocess import check_call

with open('outputfilename', 'wb', 0) as outputfile:
    check_call(shlex.split(cmd), stdout=outputfile)

Do you know what I should add to the command to prevent the subprocess from printing warning/errors to the shell ?

Just set stderr=subprocess.STDOUT to merge stdout/stderr. Or redirect stderr to os.devnull to discard the output.

jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • this helped! but `shlex.split(cmd)` was wrong. I changed it with the string cmd (`cmd = "ls dir"`) and it worked. thanks. Do you know what I should add to the command to prevent the subprocess from printing warning/errors to the shell ? – user2234234 Mar 25 '14 at 17:19
  • no, `check_call("ls dir", stdout=outputfile)` fails. You want `shlex.split("ls dir") == ["ls", "dir"]` instead. Do not use `shell=True` unless you need it. Provide explicit list argument. [Here's why](http://stackoverflow.com/q/22614070/4279) – jfs Mar 25 '14 at 17:21
  • `raise child_exception` `OSError: [Errno 2] No such file or directory` this is what i`m getting when giving it a list. – user2234234 Mar 25 '14 at 17:24
  • `cmd = command.split(" ")` `check_call(cmd, stdout=outputfile)` the second line – user2234234 Mar 25 '14 at 17:26
  • using shlex.split() is not helping. the command is calling another script. – user2234234 Mar 25 '14 at 17:30
  • @user2234234: I don't believe it. Unless you are on Windows the string argument should fail. What happens if you pass `shlex.split(command)`? If you can't tell the actual value of `command` then what its general structure (spaces, `$`, quotes `'"` are important) e.g., `"cmd -o 'arg 1' '$arg2'"` – jfs Mar 25 '14 at 17:37
  • the command is something like this: `script.pl -val_a a -val_b b -flag1 -flag2`. it has nothing special and i`m under unix. – user2234234 Mar 25 '14 at 17:46
  • `check_call(shlex.split("script.pl -val_a a -val_b b -flag1 -flag2"), stdout=outputfile)` should work if `script.pl` is in `PATH` and it has proper shebang (`#!..`) and it has proper file permissions (`chmod +x ..`) – jfs Mar 25 '14 at 17:51
  • script.pl is in PATH and it has proper shebang (#!..) and it has proper file permissions (chmod +x ..), but still not working! it`s OK man, thank you. – user2234234 Mar 25 '14 at 17:54
  • @user2234234: If it doesn't work; I want to know why. It may help others. Are sure that you haven't sneaked `shell=True`? – jfs Mar 25 '14 at 18:04
  • What happens if you call: `check_call('script.pl')` (no other arguments) in the same script? What is your Python version? – jfs Mar 25 '14 at 18:36