Django - "psycopg2.errors.ForeignKeyViolation: insert or update on table" when running "python manage.py test", but not when running single test

Question

So I made another post regarding this issue and thought I'd break it down even more to see if the good people of StackOverflow can figure out what's going on, because I can't. So I generate all this test data below using setUpTestData and things look good. I put the debugger log below to show that the test database is populated with what is expected. When I get to the test test_correct_num_posts_generated the test itself passes, but I get this weird database error. I put the full error output below. I've gotten this over many tests. I heard it could be related to a Django bug where the test DB doesn't get properly torn down. This is a weird issue, because when I run python manage.py test cheers.test.ExploreTest it passes and python manage.py test cheers.test.ExploreTest.test_correct_num_posts_generated it passes, but when I run python manage.py test I get the error shown below. What is going on?

Test.py. PostFactory is a dummy data generator.

@classmethod
    def setUpTestData(cls) -> None:
        cls.num_posts = 45
        cls.health_cate = 'Health'
        cls.fitness_cate = 'Fitness'
        cls.relationship_cate = 'Relationship'
        cls.goal_categories_name_list = [cls.health_cate, cls.fitness_cate, cls.relationship_cate]

        cls.user = create_test_user_in_DB()
        cls.access_token = get_test_user_access_token()
        for name in cls.goal_categories_name_list:
            goal_category_obj = GoalCategory.objects.create(name=name, emoji='Some URL')
            goal_obj = Goal.objects.create(creator=cls.user, goal_category=goal_category_obj)
            if name == cls.relationship_cate:
                cls.relationship_join_goal = JoinGoal.objects.create(joiner=cls.user, goal=goal_obj,
                                                                     status=GoalStatus.ONGOING)
            else:
                JoinGoal.objects.create(joiner=cls.user, goal=goal_obj, status=GoalStatus.ONGOING)

        for i in range(cls.num_posts):
            PostFactory()

Debugger print out at the end of setUpTestData. This all gets printed as well when I get to test_correct_num_posts_generated. >>> is the lines that I input and below is the print out.

>>>User.objects.all()
<QuerySet [<User 6badb4b8-33ba-4bb9-aa9a-2e3afb359960>]>
>>>GoalCategory.objects.all()
<QuerySet [<GoalCategory: GoalCategory object (Health)>, <GoalCategory: GoalCategory object (Fitness)>, <GoalCategory: GoalCategory object (Relationship)>]>
>>>len(Post.objects.all())
45
>>>JoinGoal.objects.all()
<QuerySet [<JoinGoal e7c4e4fa-3592-4ad9-b93a-245694a5e384>, <JoinGoal 7d637523-67db-4c59-aea6-37a8b43f0dd3>, <JoinGoal cbb954b7-332e-47f4-82e6-d2c77d874be5>]>

Failing test in Test.py

def test_correct_num_posts_generated(self):
    self.assertTrue(len(Post.objects.all()), self.num_posts)

self.assertTrue(len(Post.objects.all()), self.num_posts) passes, but this test is throwing the notorious error I mentioned:

======================================================================
ERROR: test_correct_num_posts_generated (cheers.test.ExploreTests.ExploreFeedTest.ExploreFeedTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\cheers\lib\site-packages\django\db\backends\utils.py", line 82, in _execute
    return self.cursor.execute(sql)
psycopg2.errors.ForeignKeyViolation: insert or update on table "cheers_post" violates foreign key constraint "cheers_post_join_goal_id_da1e6957_fk_cheers_joingoal_uuid"
DETAIL:  Key (join_goal_id)=(5c05a7d2-ff3c-4f1e-95d9-277ae164dabb) is not present in table "cheers_joingoal".


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\cheers\lib\site-packages\django\test\testcases.py", line 284, in _setup_and_call
    self._post_teardown()
  File "C:\ProgramData\Anaconda3\envs\cheers\lib\site-packages\django\test\testcases.py", line 1006, in _post_teardown
    self._fixture_teardown()
  File "C:\ProgramData\Anaconda3\envs\cheers\lib\site-packages\django\test\testcases.py", line 1248, in _fixture_teardown
    connections[db_name].check_constraints()
  File "C:\ProgramData\Anaconda3\envs\cheers\lib\site-packages\django\db\backends\postgresql\base.py", line 285, in check_constraints
    cursor.execute('SET CONSTRAINTS ALL IMMEDIATE')
  File "C:\ProgramData\Anaconda3\envs\cheers\lib\site-packages\django\db\backends\utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "C:\ProgramData\Anaconda3\envs\cheers\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "C:\ProgramData\Anaconda3\envs\cheers\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "C:\ProgramData\Anaconda3\envs\cheers\lib\site-packages\django\db\utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "C:\ProgramData\Anaconda3\envs\cheers\lib\site-packages\django\db\backends\utils.py", line 82, in _execute
    return self.cursor.execute(sql)
django.db.utils.IntegrityError: insert or update on table "cheers_post" violates foreign key constraint "cheers_post_join_goal_id_da1e6957_fk_cheers_joingoal_uuid"
DETAIL:  Key (join_goal_id)=(5c05a7d2-ff3c-4f1e-95d9-277ae164dabb) is not present in table "cheers_joingoal".

@LordElrond unfortunately the GitHub repo is private it requires access to an AWS server so has secret keys — , Dec 08 '21 at 19:55
You already tried moving all of `setUpTestData` to `setUp` (without `@classmethod`), correct? — Lord Elrond, Dec 08 '21 at 20:25
Can you narrow down the test suit to 2-3 tests that pass in isolation, but fail together? I would start by commenting out tests 1 by 1, until the entire suite passes, then you found at least 1 of the tests in question. — Lord Elrond, Dec 08 '21 at 20:28
yea so what's happening is I have a model called `Post`. If I have more than one TestCase that operates on that table in the test suite only the first TestCase that affects `Post` passes the other all fail. — , Dec 09 '21 at 01:07
The setUp function is in the same class as test_correct_num_posts_generated? — Luiz, Dec 09 '21 at 17:28
Can you post the actual test class definition and any of the setup functions in the test? — Titus P, Dec 13 '21 at 15:42

nenadp · Answer 1 · 2021-12-08T01:34:43.853

1

Since test is passing on test cheers.test.ExploreTest but not on test, there is definitely some test from some other package messing up the data.

Not sure if this will solve it, but since you are using Django 3.2, there is a small note in the docs:

Changed in Django 3.2:

Objects assigned to class attributes in setUpTestData() must support creating deep copies with copy.deepcopy() in order to isolate them from alterations performed by each test methods. In previous versions of Django these objects were reused and changes made to them were persisted between test methods.

Copying model instance using copy.deepcopy() doesn't work as expected, additional step is required to create a new instance. I can't find it anywhere in the docs but I used code from this answer multiple times (and it works on Django 3.2 with Postgres):

from copy import deepcopy
old_obj = deepcopy(obj)
old_obj.id = None
old_obj.save()

If this is the case, I'd just go with setUp instead of setUpTestData, because it recreates test database before each test.
It is slower and you might have to change specific data for some tests, but it works great and is easier to maintain.

edited Dec 08 '21 at 01:34

answered Dec 08 '21 at 01:23

nenadp

798
1
7
15

I tried `setUp` before, but I didn't know about this `deepcopy` I'll take a look – Dec 08 '21 at 04:27
also I'm not trying to create copies of data from a previous test case. – Dec 08 '21 at 04:31
`setUp` gave the same errors unfortunately – Dec 08 '21 at 04:34
I know, but it seems like django is copying data somewhere under the hood. That's unfortunate, did you replace all `setUpTestData` with `setUp` or just that one? This is now pure guessing, I'll bookmark the question in case someone comes with the solution – nenadp Dec 08 '21 at 09:43
what stands out to you that makes you believe Django is copying data? – Dec 08 '21 at 13:11
I tried replacing all `setUpTestData` functions with `setUp` didn't seem to fix it. I'm not sure how tearing down the DB and booting it up each function would do something different than just generating the DB each TestCase. Obviously there's a lot under the Django hood I'm not aware of, but I'd be surprised if that fixed it. I'd be very happy this is an annoying problem, but also surprised. – Dec 08 '21 at 13:14

score 0 · Accepted Answer · answered Dec 13 '21 at 15:44

Managed to find a solution. StackOverflow won't let me post duplicate answers, but I put the answer on this post.

Django - Serializer throwing "Invalid pk - object does not exist" when setting ManyToMany attribute where foreign keyed object does exist

Django - "psycopg2.errors.ForeignKeyViolation: insert or update on table" when running "python manage.py test", but not when running single test

2 Answers2