17

Using the following example from the documentation:

def combine_names(apps, schema_editor):
    Person = apps.get_model("yourappname", "Person")
    for person in Person.objects.all():
        person.name = "%s %s" % (person.first_name, person.last_name)
        person.save()

class Migration(migrations.Migration):    
    dependencies = [
        ('yourappname', '0001_initial'),
    ]    
    operations = [
        migrations.RunPython(combine_names),
    ]

How would I create and run a test against this migration, confirming that the data is migrated correctly?

Oskar Persson
  • 6,605
  • 15
  • 63
  • 124

4 Answers4

20

I was doing some google to address the same question and found an article that nailed the hammer on the nail for me and seemed less hacky than existing answers. So, putting this here in case it helps anyone else coming though.

The proposed the following subclass of Django's TestCase:

from django.apps import apps
from django.test import TestCase
from django.db.migrations.executor import MigrationExecutor
from django.db import connection


class TestMigrations(TestCase):

    @property
    def app(self):
        return apps.get_containing_app_config(type(self).__module__).name

    migrate_from = None
    migrate_to = None

    def setUp(self):
        assert self.migrate_from and self.migrate_to, \
            "TestCase '{}' must define migrate_from and migrate_to     properties".format(type(self).__name__)
        self.migrate_from = [(self.app, self.migrate_from)]
        self.migrate_to = [(self.app, self.migrate_to)]
        executor = MigrationExecutor(connection)
        old_apps = executor.loader.project_state(self.migrate_from).apps

        # Reverse to the original migration
        executor.migrate(self.migrate_from)

        self.setUpBeforeMigration(old_apps)

        # Run the migration to test
        executor = MigrationExecutor(connection)
        executor.loader.build_graph()  # reload.
        executor.migrate(self.migrate_to)

        self.apps = executor.loader.project_state(self.migrate_to).apps

    def setUpBeforeMigration(self, apps):
        pass

And an example use case that they proposed was:

class TagsTestCase(TestMigrations):

    migrate_from = '0009_previous_migration'
    migrate_to = '0010_migration_being_tested'

    def setUpBeforeMigration(self, apps):
        BlogPost = apps.get_model('blog', 'Post')
        self.post_id = BlogPost.objects.create(
            title = "A test post with tags",
            body = "",
            tags = "tag1 tag2",
        ).id

    def test_tags_migrated(self):
        BlogPost = self.apps.get_model('blog', 'Post')
        post = BlogPost.objects.get(id=self.post_id)

        self.assertEqual(post.tags.count(), 2)
        self.assertEqual(post.tags.all()[0].name, "tag1")
        self.assertEqual(post.tags.all()[1].name, "tag2")
devinm
  • 755
  • 1
  • 7
  • 17
  • 1
    That article was perfect. No need for additional packages - all native Django. We followed and it worked perfectly. Thanks @devinm – ZAR Apr 03 '20 at 15:56
  • 1
    Actually, this code is good if you *only* plan on testing data changes. The second you test schema changes, it falls short; django will complain about reading and writing models because they're out of sync with the db. The library mentioned in @sobolevn's answer handles that well, and I think that makes it a better solution. – Daniel Kaplan Jul 08 '21 at 16:43
  • @DanielKaplan I don't think they should be out of sync: the above example uses historical models (and not the actual models imported from the modules), just as you do in migrations. As far as I can see, both `setUpBeforeMigration` and the test method uses the correct version with respect to the db state. – atleta May 02 '22 at 17:49
  • 1
    For those that didn't pay attention like me, note that this requires that your migration steps are reversible. The way it works is that it reverses migrations back from the latest/current migration back to the one you specify. – Eyad Arafat Jun 27 '22 at 02:31
  • `django-test-migrations` mentioned in @sobolevn's answer fit my needs better since what I actually wanted to test was in fact the reverse migration. – Eyad Arafat Jun 27 '22 at 02:43
13

You can use django-test-migrations package. It is suited for testing: data migrations, schema migrations, and migrations' order.

Here's how it works:

from django_test_migrations.migrator import Migrator

# You can specify any database alias you need:
migrator = Migrator(database='default')

old_state = migrator.before(('main_app', '0002_someitem_is_clean'))
SomeItem = old_state.apps.get_model('main_app', 'SomeItem')

# One instance will be `clean`, the other won't be:
SomeItem.objects.create(string_field='a')
SomeItem.objects.create(string_field='a b')

assert SomeItem.objects.count() == 2
assert SomeItem.objects.filter(is_clean=True).count() == 2

new_state = migrator.after(('main_app', '0003_auto_20191119_2125'))
SomeItem = new_state.apps.get_model('main_app', 'SomeItem')

assert SomeItem.objects.count() == 2
# One instance is clean, the other is not:
assert SomeItem.objects.filter(is_clean=True).count() == 1
assert SomeItem.objects.filter(is_clean=False).count() == 1

We also have native integrations for both pytest:

@pytest.mark.django_db
def test_main_migration0002(migrator):
    """Ensures that the second migration works."""
    old_state = migrator.before(('main_app', '0002_someitem_is_clean'))
    SomeItem = old_state.apps.get_model('main_app', 'SomeItem')
    ...

And unittest:

from django_test_migrations.contrib.unittest_case import MigratorTestCase

class TestDirectMigration(MigratorTestCase):
    """This class is used to test direct migrations."""

    migrate_from = ('main_app', '0002_someitem_is_clean')
    migrate_to = ('main_app', '0003_auto_20191119_2125')

    def prepare(self):
        """Prepare some data before the migration."""
        SomeItem = self.old_state.apps.get_model('main_app', 'SomeItem')
        SomeItem.objects.create(string_field='a')
        SomeItem.objects.create(string_field='a b')

    def test_migration_main0003(self):
        """Run the test itself."""
        SomeItem = self.new_state.apps.get_model('main_app', 'SomeItem')

        assert SomeItem.objects.count() == 2
        assert SomeItem.objects.filter(is_clean=True).count() == 1
Rafael Almeida
  • 5,142
  • 2
  • 20
  • 33
sobolevn
  • 16,714
  • 6
  • 62
  • 60
  • The library is only for python3 this solution doesn't support python2 – Rafael Almeida Mar 27 '20 at 10:31
  • 7
    @RafaelAlmeida That is a good reminder that any serious application that is still under some development ought to be, um, migrated to Python 3 by now. – Lutz Prechelt Nov 12 '21 at 16:56
  • 2
    @RafaelAlmedia, the package supports Django 2.2, 3.1, 3.2 and 4.0 - None of these Django versions support Python 2, so Python 2 support would be redundant. It's also worth noting that any Django version that supports Python 2 is end of life. – JGC Jan 24 '22 at 18:00
  • I get error - relation already exists . – Aseem Jan 06 '23 at 22:21
7

EDIT:

These other answers make more sense:

ORIGINAL:

Running your data-migration functions (such as combine_names from the OP's example) through some basic unit-tests, before actually applying them, makes sense to me too.

At first glance this should not be much more difficult than your normal Django unit-tests: migrations are Python modules and the migrations/ folder is a package, so it is possible to import things from them. However, it took some time to get this working.

The first difficulty arises due to the fact that the default migration file names start with a number. For example, suppose the code from the OP's (i.e. Django's) data-migration example sits in 0002_my_data_migration.py, then it is tempting to use

from yourappname.migrations.0002_my_data_migration import combine_names

but that would raise a SyntaxError because the module name starts with a number (0).

There are at least two ways to make this work:

  1. Rename the migration file so it does not start with a number. This should be perfectly fine according to the docs: "Django just cares that each migration has a different name." Then you can just use import as above.

  2. If you want to stick to the default numbered migration file names, you can use Python's import_module (see docs and this SO question).

The second difficulty arises from the fact that your data-migration functions are designed to be passed into RunPython (docs), so they expect two input arguments by default: apps and schema_editor. To see where these come from, you can inspect the source.

Now, I'm not sure this works for every case (please, anyone, comment if you can clarify), but for our case, it was sufficient to import apps from django.apps and get the schema_editor from the active database connection (django.db.connection).

The following is a stripped-down example showing how you can implement this for the OP example, assuming the migration file is called 0002_my_data_migration.py:

from importlib import import_module
from django.test import TestCase
from django.apps import apps
from django.db import connection
from yourappname.models import Person
# Our filename starts with a number, so we use import_module
data_migration = import_module('yourappname.migrations.0002_my_data_migration')


class DataMigrationTests(TestCase):
    def __init__(self, *args, **kwargs):
        super(DataMigrationTests, self).__init__(*args, **kwargs)
        # Some test values
        self.first_name = 'John'
        self.last_name = 'Doe'
        
    def test_combine_names(self):
        # Create a dummy Person
        Person.objects.create(first_name=self.first_name,
                              last_name=self.last_name, 
                              name=None)
        # Run the data migration function
        data_migration.combine_names(apps, connection.schema_editor())
        # Test the result
        person = Person.objects.get(id=1)
        self.assertEqual('{} {}'.format(self.first_name, self.last_name), person.name)
        
djvg
  • 11,722
  • 5
  • 72
  • 103
  • If you just want to test your data migration then this snippet works great! – Erik Kalkoken Feb 26 '20 at 17:45
  • If, at some later time, you rename `Person.first_name` into `Person.firstname`, this test will stop working _although_ the migration and the remainder of the app are still correct. This kind of test is a one-shot thing that should be deleted once the migration has been committed (preferably along with the test). – Lutz Prechelt Nov 12 '21 at 17:02
  • @LutzPrechelt: You're right. That's why I tried sending people towards the other answers. ;-) – djvg Nov 12 '21 at 18:58
0

You could add a crude if statement to a prior migration that tests if the test suite is running, and adds initial data if it is -- that way you can just write a test to check if the objects are in the final state you want them in. Just make sure your conditional is compatible with production, here's an example that would work with python manage.py test:

import sys
if 'test in sys.argv:
    # do steps to update your operations

For a more "complete" solution, this older blog post has some good info and more up-to-date comments for inspiration:

https://micknelson.wordpress.com/2013/03/01/testing-django-migrations/#comments

whp
  • 1,406
  • 10
  • 10