How to load fixtures in Django south migrations properly?

Question

I am using Django 1.5b1 and south migrations and life has generally been great. I have some schema updates which create my database, with a User table among others. I then load a fixture for ff.User (my custom user model):

def forwards(self, orm):
        from django.core.management import call_command
        fixture_path = "/absolute/path/to/my/fixture/load_initial_users.json"
        call_command("loaddata", fixture_path)

All has been working great until I have added another field to my ff.User model, much further down the migration line. My fixture load now breaks:

DatabaseError: Problem installing fixture 'C:\<redacted>create_users.json':
Could not load ff.User(pk=1): (1054, "Unknown column 'timezone_id' in 'field list'")

Timezone is the field (ForeignKey) which I added to my user model.

The ff.User differs from what is in the database, so the Django ORM gives up with a DB error. Unfortunately, I cannot specify my model in my fixture as orm['ff.User'], which seems to be the south way of doing things.

How should I load fixtures properly using south so that they do not break once the models for which these fixtures are for gets modified?

Answer 1

I found a Django snippet that does the job!

https://djangosnippets.org/snippets/2897/

It load the data according to the models frozen in the fixture rather than the actual model definition in your apps code! Works perfect for me.

Answer 2

I proposed a solution that might interest you too:

https://stackoverflow.com/a/21631815/797941

Basicly, this is how I load my fixture:

from south.v2 import DataMigration
import json

class Migration(DataMigration):

    def forwards(self, orm):
        json_data=open("path/to/your/fixture.json")
        items = json.load(json_data)
        for item in items:
            # Be carefull, this lazy line won't resolve foreign keys
            obj = orm[item["model"]](**item["fields"])
            obj.save()

        json_data.close()

Answer 3

This was a frustrating part of using fixtures for me as well. My solution was to make a few helper tools. One which creates fixtures by sampling data from a database and includes South migration history in the fixtures.

There's also a tool to add South migration history to existing fixtures.

The third tool checks out the commit when this fixture was modified, loads the fixture, then checks out the most recent commit and does a south migration and dumps the migrated db back to the fixture. This is done in a separate database so your default db doesn't get stomped on.

The first two can be considered beta code, and the third please treat as usable alpha, but they're already being quite helpful to me.

Would love to get some feedback from others: git@github.com:JivanAmara/django_fixture_tools.git Currently, it only supports projects using git as the RCS.

Answer 4

The most elegant solution I've found is here where by your app model's get_model function is switched out to instead supply the model from the supplied orm. It's then set back after the fixture is applied.

from django.db import models
from django.core.management import call_command

def load_fixture(file_name, orm):
    original_get_model = models.get_model

    def get_model_southern_style(*args):
        try:
            return orm['.'.join(args)]
        except:
            return original_get_model(*args)

    models.get_model = get_model_southern_style
    call_command('loaddata', file_name)
    models.get_model = original_get_model

You call it with load_fixture('my_fixture.json', orm) from within you forwards definition.

Answer 5

Generally South handles migrations using forwards() and backwards() functions. In your case you should either:

• alter the fixtures to contain proper data, or
• import fixture before migration that breaks it (or within the same migration, but before altering the schema),

In the second case, before migration adding (or, as in your case, removing) the column, you should perform the migration that will explicitly load the fixtures similarly to this (docs):

def forwards(self, orm):
    from django.core.management import call_command
    call_command("loaddata", "create_users.json")

I believe this is the easiest way to accomplish what you needed. Also make sure you do not do some simple mistakes like trying to import data with new structure before applying older migrations.

Answer 6

Reading the following two posts has helped me come up with a solution:

http://andrewingram.net/2012/dec/common-pitfalls-django-south/#be-careful-with-fixtures

http://news.ycombinator.com/item?id=4872596

Specifically, I rewrote my data migrations to use output from 'dumpscript'

I needed to modify the resulting script a bit to work with south. Instead of doing

from ff.models import User

I do

User = orm['ff.User']

This works exactly like I wanted it to. Additionally, it has the benefit of not hard-coding IDs, like fixtures require.