Django test with setUpTestData doesn't keep changes between tests

Question

classmethod TestCase.setUpTestData()

The class-level atomic block described above allows the creation of initial data at the class level, once for the whole TestCase.

[...]

Be careful not to modify any objects created in setUpTestData() in your test methods. Modifications to in-memory objects from setup work done at the class level will persist between test methods.

Cit. django.test.TestCase.setUpTestData

Consider this example:

class FoobarTest(TestCase):

    @classmethod
    def setUpTestData(cls):
        cls.post = Post()
        cls.post.stats = {}
        cls.post.save()

    def setUp(self):
        self.post.refresh_from_db()

    def test_foo(self):
        self.post.stats['foo'] = 1
        self.post.save()
        self.assertEquals(self.post.stats, {'foo': 1}) # this should fail

    def test_bar(self): # this run first cause alphabetical order
        self.post.stats['bar'] = 1
        self.post.save()
        self.assertEquals(self.post.stats, {'bar': 1})

Since

Modifications to in-memory objects from setup work done at the class level will persist between test methods

I expect one of the two test methods to fail cause the post object will have also a different property and the equality should fail.

But this test pass without problem.

If i force the execution order it actually behave like expected:

class FoobarTest(TestCase):

    @classmethod
    def setUpTestData(cls):
        cls.post = Post()
        cls.post.stats = {}
        cls.post.save()

    def setUp(self):
        self.post.refresh_from_db()

    def _foo(self):
        self.post.stats['foo'] = 1
        self.post.save()
        self.assertEquals(self.post.stats, {'foo': 1})

    def _bar(self):
        self.post.stats['bar'] = 1
        self.post.save()
        self.assertEquals(self.post.stats, {'bar': 1})

    def test_foo_bar(self):
        self._foo()
        self._bar() # this fail

The question is:

in the first example, are the test methods running in some kind of parallelism? Can they actually fails under some time-related coincidences?

The Django docs about order in which tests are executed doens't talk about test methods.

In this post i found that the order of test methods can be changed, but is one after another, without parallelism, unless i use python manage.py test --parallel.

To anyone reading this today, Django 3.2+ now make a clean copy of your database, so they are clearly separated https://docs.djangoproject.com/en/3.2/topics/testing/tools/#django.test.TestCase.setUpTestData — Jean Bouvattier, Mar 23 '22 at 08:57
setUpTestData is called for each test run in case of database with not transaction support e.g. MySQL, this would also explain this behaviour — Jean Bouvattier, Mar 23 '22 at 08:59

niccord · Answer 1 · 2019-10-04T11:44:49.547

This is my test, using memory instead of database:

from django.test import TestCase

class Post(object):
    def __init__(self):
        self.stats = {}

    def refresh_from_db(self):
        pass

    def save(self):
        pass


class FoobarTest(TestCase):

    @classmethod
    def setUpTestData(cls):
        cls.post = Post()
        cls.post.stats = {}

    def setUp(self):
        self.post.refresh_from_db()

    def test_foo(self):
        self.post.stats['foo'] = 1
        self.post.save()
        self.assertEquals(self.post.stats, {'foo': 1})  # this should fail

    def test_bar(self):  # this run first cause alphabetical order
        self.post.stats['bar'] = 1
        self.post.save()
        self.assertEquals(self.post.stats, {'bar': 1})

And this is my result:

    self.assertEquals(self.post.stats, {'foo': 1})  # this should fail
AssertionError: {'bar': 1, 'foo': 1} != {'foo': 1}
- {'bar': 1, 'foo': 1}
+ {'foo': 1}

Changing the method foo to afoo, the result is as expected:

    self.assertEquals(self.post.stats, {'bar': 1})
AssertionError: {'foo': 1, 'bar': 1} != {'bar': 1}
- {'bar': 1, 'foo': 1}
+ {'bar': 1}

So maybe this is a database issue somehow?

In your assertion you are checking in-memory objects: if save fails, refresh_from_db returns the starting empty object, so the next check is passed.

Try getting data back after save, then check them.

In the example if `save` fails i'm expecting to receive an exception. Also in the "real" code i checked this out and there aren't saving errors. — Andrea Franchini, Oct 04 '19 at 12:17
I think receiving an exception and all this you are experiencing belongs to the model implementation or with some database/driver issue. — niccord, Oct 04 '19 at 13:02

Django test with setUpTestData doesn't keep changes between tests

1 Answers1