3

Calling subprocess.check_call() allows to specify a file object for stdout, but before writing the data to a file, I would like to modify them on a line-by-line base.

I currently redirect the output to a temporary file (created by tempfile.TemporaryFile(). After check_call finished, I read that temporary file line-by-line, do the modifications and write the final output file.

Since the output is large, a pure in-memory-solution is not feasible and I would like to modify the data on-the-fly and write to the final output file directly.

Does anyone know how to achieve that?

Daniel
  • 31
  • 1
  • 2

2 Answers2

2
def check_call_modify(command, modifier_function, output_file)
    p = subprocess.Popen(command, stdout=subprocess.PIPE)
    for line in p.stdout:
        line = modifier_function(line)
        output_file.write(line)    
    p.wait()
    if p.returncode:
        raise subprocess.CalledProcessError(p.returncode, command)
    return p.returncode

to use it pass a function to modify each line, and the file. Dumb example below will save the result of ls -l in uppercase to listupper.txt:

with open('listupper.txt', 'w') as f:
    check_call_modify(['ls', '-l'], operator.methodcaller('upper'), f)
nosklo
  • 217,122
  • 57
  • 293
  • 297
-1

Python is duck-typed, so you could always wrap your file object before passing it to check_call.

This answer has an example of doing it for write() but, to be thorough, you'd probably also want to wrap writelines().

Community
  • 1
  • 1
ssokolow
  • 14,938
  • 7
  • 52
  • 57
  • -1: no, it won't work, since subprocess needs a object with a `.fileno()` method, and writes directly to the file descriptor returned by that. In fact, the file descriptor is passed directly to the subprocess, so python is not involved in the writing. – nosklo Mar 30 '11 at 21:45
  • Odd. I thought I remembered being able to pass in file-like objects without a `.fileno()`. Oh well. My mistake, I guess. – ssokolow Mar 31 '11 at 12:05