3

I'm running a WSGI server and part of the API I'm writing returns some (rather large) files along with meta-data about them. I'd like to tar/gzip the files together both to conserve bandwidth and so only one file has to be downloaded. Since WSGI lets you return an iterable object, I'd like to return an iterable that returns chunks of the tar.gz file as it's produced.

My question is what's a good way to tar/gzip files together in Python in a way that's amenable to streaming the output back to the user?

EDIT:

To elaborate on my response to Oben Sonne below, I'll have a function such as:

def iter_file(f,chunk=32768): return iter(lambda: f.read(chunk), '')

Which will let me specify a chunk size to return from the file when returning it to the WSGI server.

Then it's a simple matter of:

return iter_file(subprocess.Popen(["tar", "-Ocz"] + files, stdout=subprocess.PIPE).stdout)

or, if I want to return a file: 

return iter_file(open(filename, "rb"))
gct
  • 14,100
  • 15
  • 68
  • 107

1 Answers1

2

The bz2 module provides sequential compressing. And it seems the zlib package can compress data sequentially too. So with these modules you could:

  1. tar your files (shouldn't take that long),
  2. read the archive iteratively in binary mode,
  3. pass read chunks to a sequential compression function, and
  4. yield the compressed output of these functions so it may be consumed iteratively by some other component (WSGI)

AFAIK Python's tar-API does not support sequential tar'ing (correct me if I'm wrong). But if your files are so large that you really need to tar sequentially, you could use the subprocess module to run tar on the command line and read its standard output in chunks. In that case you could also use the tar command to compress your data. Then you only had to read the stdout of your subprocess and yield read chunks.

Community
  • 1
  • 1
Oben Sonne
  • 9,893
  • 2
  • 40
  • 61
  • I think this is what I'm going to do, I can basically return "subprocess.Popen(shlex.split("tar -Ocz "), stdout=subprocess.PIPE).stdout" to the WSGI server and it will do what I want. – gct Jun 21 '11 at 20:44
  • @gct: Why go via shlex? Just use `subprocess.Popen(["tar", "-Ocz",] + files, [...])` where files is a list of filenames. – phant0m Jun 21 '11 at 21:34
  • Just how the code I had lying around to use subprocess did it, I used your way up above though. – gct Jun 21 '11 at 21:38
  • @gct: Looks good, *subprocess* probably is the most simple solution, I'd prefer it too. – Oben Sonne Jun 22 '11 at 06:00