4

I've got a django app that I'm moving to rackspace. I have a model that uses FileFields and I'm using the django-storages library s3/boto backend. I want to use cloudfiles for storage, and I need to be able to serve up the old s3 content.

On a template page where I provide links to the files, I do this:

href="{{ static_url }}{{ article.code_archive_file}}"

static_url is set from the view and equals settings.STATIC_URL. Clearly this isn't going to work since settings.STATIC_URL is going to change when I switch from s3.

Do I need to write a script to migrate all of my s3 files by hand to cloudfiles and also go through and update all the FileFields in my tables? (ugh). I'd rather be able to change out the storages backend and leave the old material in the s3 bucket.

If I do need to migrate the files and the fields, has someone already written a script for that?

bonus question: What best-practices didn't I follow when doing this? I've only been using django for about half a year now.

skm
  • 53
  • 4
  • 2
    Off the cuff, I recommend using [django-cumulus](https://github.com/richleland/django-cumulus) to interact with CloudFiles. For migrating, I highly recommend using [wonton](https://github.com/rackerlabs/wonton) to transfer files over quickly with gevent. – Kyle Kelley Jan 03 '14 at 20:28
  • As an alternative to what @KyleKelley suggests, you can adding a try-except-try pattern that will first attempt to grab the file from S3, if it gets a 404, it tries from CloudFiles and if that fails then the file isn't anywhere. We had to do something similar with S3 and Google Cloud Storage. – rdodev Jan 03 '14 at 20:35
  • Also, I should say that I meant pure file migration, not the database migration. That part sounds time-consuming (but scriptable). – Kyle Kelley Jan 03 '14 at 20:41
  • 1
    Oh. Just started looking at django-storages. It looks like the CloudFiles backend is way out of date (it references a Mosso CloudFiles module that has since been deprecated). If you continue using django-storages I recommend using the libcloud backend. – Kyle Kelley Jan 03 '14 at 20:59
  • 1
    Thanks for the pointer to django-cumulus. It looks like it might be easier to do some of the things I wanted -- like use multiple containers -- that I wasn't sure how to accomplish with django-storages. Actually, thanks also for pointing me to the libcloud backend. If I stick with django-storages I will probably switch to that. – skm Jan 03 '14 at 21:47

1 Answers1

0

Option A: You can use a bit different paths on old and new storages and have something to decide from where file should be served. Something could be custom storage class, I think: you create storage class inherited from CloudFilesStorage, and in overrided url method it will either call super (if file is in the new storage) or call url method from S3BotoStorage (if file is in the old storage).

Option B: You can move your files to cloudfiles. I don't think you actually need to change anything in DB after that if you keep paths similar. Maybe you even don't need a script, just download files and then upload then (s3cmd can be used for downloading, cloudfiles probably got some tool for uploading too).

About best practices: you should've used media_url and DEFAULT_FILE_STORAGE instead of static_url and static storage. It's a good idea to keep project's static files (css/js/icons) and uploaded media separated.

Ivan Anishchuk
  • 487
  • 3
  • 16
  • My media_url is built based on static_url and both would have a url based on the s3 bucket, due to how django-storages works. I'd prefer to use two buckets (or containers, in cloudfiles parlance). – skm Jan 16 '14 at 20:14
  • Ok. Why don't you do that? – Ivan Anishchuk Jan 18 '14 at 08:16
  • I am using the boto s3 backend for django-storages, which does not support multiple buckets as far as I can tell. I think the solution will be to switch to django-cumulus or to see if the libcloud backend for django storages will allow multiple buckets/containers. The docs are unclear, so I will need to dig. – skm Jan 22 '14 at 15:24
  • 2
    Never heard of it, static and default storages could be not only on different buckets, they could even use different backends. Take a look at [this project](https://github.com/jamstooks/django-s3-folder-storage) as an example – for different bucket you gonna need to use different bucket names in STATIC_URL/MEDIA_URL and pass those bucket names to storage classes (inheriting them and override bucket name). Some additional information could be found in [this answer](https://github.com/jamstooks/django-s3-folder-storage) – Ivan Anishchuk Jan 23 '14 at 18:23