using gzip file as stdin for commands executed using subprocess.call

Question

I have a python script, executing multiple commands using subprocess.call(). I need to pass data from a gzipped file to one of those commands using stdin, but no matter what I do, the command apparently gets the gzipped data.

This is what I think should work:

import gzip
from subprocess import call

in_fname = 'test.gz'
out_fname = 'test.txt'

gz = gzip.open(in_fname, 'rb')
txt = open(out_fname, 'w')

call(['cat'], stdin=gz, stdout=txt)

But at the end, the 'test.txt' is compressed and has exactly the same size as the gzipped input file.

If I call gz.read() then I get the correct decompressed data, as expected. What do I need to do to use the gzipped file as stdin?

Answer 1

After doing a bit of research, the root of the problem stems from the fact that your operating system has no idea that the filehandle for the gzipped file is anything special. Basically, gzip provides a "file-like" interface but the subprocess (cat in this case) has no idea that this is a special file that needs to be unzipped. Therefore, it just reads the file byte for byte and prints out the gibberish it reads.

My next idea was to read the whole file in python (which knows it's compressed and will unzip it) and then pass the string to the subprocess. I messed around with wrapping the unzipped contents in a StringIO object but that turns out not to work. Another answer (Use StringIO as stdin with Popen) mentioned a slightly different call to subprocess:

import gzip
from subprocess import Popen, PIPE

in_fname = 'test.gz'
out_fname = 'test.txt'

with gzip.open(in_fname, 'rb') as f:
  gz = f.read()
txt = open(out_fname, 'w')


process = Popen(['cat'], stdin=PIPE, stdout=txt)
process.communicate(gz)

Which works. Note that this requires reading the whole file into memory which may be a problem for really big files.