4

I have a django app on heroku which serves the static files from amazon s3 bucket. I use boto library and followed the guide on the website. What can I do to speed up the file transfers?

Some of the code:

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'xxxx'
AWS_SECRET_ACCESS_KEY = 'xxxx'
AWS_STORAGE_BUCKET_NAME = 'boxitwebservicebucket'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATIC_URL = 'http://' + AWS_STORAGE_BUCKET_NAME + '.s3.amazonaws.com/'

the view

class GetFileContent(View):
    def get(self,request, userName, pk):
        user = User.objects.filter(username = userName)
        filtered = Contentfile.objects.filter(pk = pk, published=True, file_owner = user)
        data = filtered[0].content
        filename = filtered[0].title + "." + filtered[0].file_type
        response = HttpResponse(data, content_type='application/force-download')
        response['Content-Disposition'] = 'attachment; filename="{}"'.format(filename)
        return response
        pass

I suspect django is serving the file even though it sits on the s3 server, How can I direct the user to the s3 link directly?

Yoav Schwartz
  • 2,017
  • 2
  • 23
  • 43
  • I would recommend serving files from S3 via CloudFront instead. – Brandon Taylor Dec 20 '13 at 12:51
  • Currently, Im in demo stages and I would like to keep everything free. However Im not sure that's the problem, I'm beginning to suspect that actually even though I'm using amazon s3 storage, Django is serving the files instead of amazon because when I look at my network traffic, I can't see any links leading to amazon. – Yoav Schwartz Dec 20 '13 at 13:07
  • CloudFront would only cost you $0.12 for the first 10 TB out per month. That's pretty affordable :). If your network traffic says that your files are being served by Django, that's a different issue altogether. – Brandon Taylor Dec 20 '13 at 13:10
  • Yeah, It's the first time I'm trying to serve files using Django, and I can find a lot of documentation on how to set up the static files for the site to be served with django, but non about how to serve downloadable file, I would have guessed it would be more common of a problem. – Yoav Schwartz Dec 20 '13 at 13:12
  • Have you seen: http://stackoverflow.com/questions/1156246/having-django-serve-downloadable-files ? – Brandon Taylor Dec 20 '13 at 13:17
  • I have, I'm not sure how to implement the answers though. Non are talking about s3 or s3boto – Yoav Schwartz Dec 20 '13 at 13:24
  • Is not the only thing different about your code compared to the example the path to the file? – Brandon Taylor Dec 20 '13 at 14:05
  • 2
    If you look here at django-storages, you will see that the url function accepts a header param. https://bitbucket.org/david/django-storages/src/3cb6a9b1308cfb110cad5aa92bb6db6f7de110c4/storages/backends/s3boto.py#cl-457 You should try to generate a url for that file using django storages, making sure to pass the Content-Disposition 'attachment' in the headers, and then return a 302 redirect to that url. If all goes well the file will be served from Amazon with the right header, forcing the browser to download instead of open the url. – Pierre Drescher Jan 07 '14 at 21:02
  • It's actually not a browser, It's a web service for an app, so I have no problem just making it download without sending attachment. actually I arrived at the same solution just a couple of hours before you posted, I just return the 'content.url' just like you suggested, only in JSON and than I start the download from the device to that link. Anyways, this is the correct answer in my opinion, so if you can refractor your comment to a more elaborated answer with sample code I would love to give you a well deserved bounty ;) – Yoav Schwartz Jan 08 '14 at 08:52
  • @pdr Please post an answer to the question with some sample code and further explanation so I can give you the bounty. – Yoav Schwartz Jan 09 '14 at 14:57
  • I answered a similar question to this over here: http://stackoverflow.com/questions/12975228/download-files-from-amazon-s3-with-django/22890568#22890568 – so_ Apr 06 '14 at 06:18

1 Answers1

4

Here is how I do it - It does not feel slow to me:

models.py:

class Document(models.Model):
    id = UUIDField(primary_key = True)
    extension = models.CharField(max_length = 5)
    created_on = CreationDateTimeField()
    labels = models.ManyToManyField(Label)

    def url(self, bucket):
        url = get_s3_url(bucket, '' + str(self.id) + str(self.extension) + '')
            return 'https' + url[4:]

views.py:

import urllib2


@login_required
def view(request, document_id):
    document = Document.objects.get(id = document_id)

    response_file = urllib2.urlopen(document.url(request.user.profile.aws_documents_bucket_name))

    response = HttpResponse(response_file.read(), mimetype = document.mimetype)
    response['Content-Disposition'] = 'inline; filename=' + str(document.id) + document.extension

    return response

utils.py:

from boto.s3.connection import S3Connection
from boto.s3.key import Key
from django.conf import settings


def get_s3_url(bucket, filename):
    s3 = S3Connection(settings.AWS_ACCESS_KEY_ID, settings.AWS_SECRET_ACCESS_KEY)
    key = s3.create_bucket(bucket).get_key('' + filename + '')
    return key.generate_url(3600, "GET", None, True, True) # This gives an authenticated url available for only a short time period (by design)

My individual users or groups of users have designated buckets referenced in the profile object. AWS Credentials stored in settings.py.

Mathieu Steele
  • 693
  • 4
  • 7