Downloading and zipping files from amazon

Question

I'm currently storing all my photos on amazon s3 and using django for my website. I want a to have a button that allows users to click it and have all their photos zipped and returned to them.

I'm currently using boto to interface with amazon and found that I can go through the entire bucket list / use get_key to look for specific files and download them

After this I would need temporarily store them, then zip and return.

What is the best way to go about doing this?

Thanks

[possible duplicate](http://stackoverflow.com/questions/908258/generating-file-to-download-with-django) — Lynob, Feb 29 '12 at 09:56
close except I still need to download all the images first then deal with the zipping / returning — mirugai, Feb 29 '12 at 10:46
are your images located in one folder in s3? or are they located in many folders? also does it have to be django? i would like to use shell scripting — Lynob, Feb 29 '12 at 11:58
yes the button allows users to download their own photos. Also the photos are all in hte same bucket — mirugai, Feb 29 '12 at 16:52

score 1 · Answer 1 · edited May 23 '17 at 11:58

Using python-zipstream as patched with this pull request you can do something like this:

import boto
import io
import zipstream
import sys


def iterable_to_stream(iterable, buffer_size=io.DEFAULT_BUFFER_SIZE):
    """
    Lets you use an iterable (e.g. a generator) that yields bytestrings as a
    read-only input stream.

    The stream implements Python 3's newer I/O API (available in Python 2's io
    module).  For efficiency, the stream is buffered.

    From: https://stackoverflow.com/a/20260030/729491
    """
    class IterStream(io.RawIOBase):
        def __init__(self):
            self.leftover = None

        def readable(self):
            return True

        def readinto(self, b):
            try:
                l = len(b)  # We're supposed to return at most this much
                chunk = self.leftover or next(iterable)
                output, self.leftover = chunk[:l], chunk[l:]
                b[:len(output)] = output
                return len(output)
            except StopIteration:
                return 0    # indicate EOF
    return io.BufferedReader(IterStream(), buffer_size=buffer_size)


def iterate_key():
    b = boto.connect_s3().get_bucket('lastage')
    key = b.get_key('README.markdown')
    for b in key:
        yield b

with open('/tmp/foo.zip', 'w') as f:
    z = zipstream.ZipFile(mode='w')
    z.write(iterable_to_stream(iterate_key()), arcname='foo1')
    z.write(iterable_to_stream(iterate_key()), arcname='foo2')
    z.write(iterable_to_stream(iterate_key()), arcname='foo3')
    for chunk in z:
        print "CHUNK", len(chunk)
        f.write(chunk)

Basically we iterate over the key contents using boto, convert this iterator to a stream using the iterable_to_stream method from this answer and then have python-zipstream create a zip file on-the-fly.

score 1 · Accepted Answer · edited May 23 '17 at 12:22

you can take a look at this question or at this snippet to download the file

# This is not a full working example, just a starting point
# for downloading images in different formats.

import subprocess
import Image

def image_as_png_pdf(request):
  output_format = request.GET.get('format')
  im = Image.open(path_to_image) # any Image object should work
  if output_format == 'png':
    response = HttpResponse(mimetype='image/png')
    response['Content-Disposition'] = 'attachment; filename=%s.png' % filename
    im.save(response, 'png') # will call response.write()
  else:
    # Temporary disk space, server process needs write access
    tmp_path = '/tmp/'
    # Full path to ImageMagick convert binary
    convert_bin = '/usr/bin/convert' 
    im.save(tmp_path+filename+'.png', 'png')
    response = HttpResponse(mimetype='application/pdf')
    response['Content-Disposition'] = 'attachment; filename=%s.pdf' % filename
    ret = subprocess.Popen([ convert_bin, 
                            "%s%s.png"%(tmp_path,filename), "pdf:-" ],
                            stdout=subprocess.PIPE)
    response.write(ret.stdout.read())
  return response

to create a zip follow the link that i gave you, you can also use zipimport as shown here examples are on the bottom of the page, follow the documentation for newer versions

you might also be interested in this although it was made for django 1.2, it might not work on 1.3

Downloading and zipping files from amazon

2 Answers2