Django unittesting: model.all() queryset is empty in a method running in separate thread

Question

I have a method running in a separate thread, that does some contacts matching.

I'm writing tests to check if the contacts have been synced. The test case goes something like this:

class ContactSyncTestCase(TestCase):
    fixtures = ['fix.json']

    def setUp(self):
        # get a few contacts that exist in the database to be sent for matching
        self.few_contacts = CompanyContact.objects.all().order_by('?')[:5].values_list('contact_number',flat=True)

    def test_submit_contacts(self):

        # Pick up a random user
        user = User.objects.all().order_by('?')[0]

        # Get API Key for that user
        key = ApiKey.objects.get(user=user).key

        # The url that submits contacts for matching, returns a matching key immediately and starts a separate thread to sync contacts
        sync_request_url = '/sync_contacts/?username=%s&api_key=%s'%(user.username,key)
        sync_request_response = self.client.post(path=sync_request_url,
                                    data=json.dumps({"contacts":','.join(self.few_contacts)}),
                                    content_type="application/json")

        # Key that is used to fetch the status of contacts syncing and returns a json if contacts are matched
        key = sync_request_response.content

        # At this point, the other thread is doing the task of syncing inside the method mentioned next

        # Hence I put the test on pause so that it does not fail and exit
        try:
            while True:
                time.sleep(100)
        except KeyboardInterrupt:
            pass

The async method that matches the numbers starts something like this:

def match_numbers(key, contacts, user):
    # Get all contacts stored in the system
    """

    :param key:
    :param contacts:
    :param user:
    """
    import pdb;pdb.set_trace()
    system_contacts = CompanyContact.objects.all().values_list('contact_number', flat=True)

Now the weird issue here is that:

CompanyContact.objects.all().values_list('contact_number', flat=True)

Returns an empty queryset while testing. However, during runtime it works fine.

For that matter, any query (including User model) returns an empty queryset.

Any ideas why?

EDIT:

Turns out that inheriting from TrasactionTestCase solves this issue. I still have my doubts and I dug up more for the same.

My database's default transaction level is Repeatable Read.

Reading from this post,

REPEATABLE READ (default) : ensure that is a transaction issues the same SELECT twice, it gets the same result both times, regardless of committed or uncommitted changes made by other transactions. In other words, it gets a consistent result from different executions of the same query. In some database systems, REPEATABLE READ isolation level allows phantoms, such that if another transaction inserts new rows,in the interval between the SELECT statements, the second SELECT will see them. This is not true for InnoDB; phantoms do not occur for the REPEATABLE READ level.

Summary of this: I should still have got the existing records, which I didn't.

Try using TransactionTestCase, see http://stackoverflow.com/a/10949616/1423473 — erthalion, Feb 21 '14 at 14:29
That works! TransactionTestCase. However, I'm not sure why would it not throw an error but return an empty queryset. — Gaurav Toshniwal, Feb 21 '14 at 14:34
It returns an empty queryset, because it trying to connect directly to db while the all test data kept in the memory. — erthalion, Feb 21 '14 at 14:40
I didn't get this part. I would be connecting to Test DB only right? Is there a migration of data from the test DB to memory? Any part of the documentation that explains this? — Gaurav Toshniwal, Feb 21 '14 at 14:53

score 3 · Answer 1 · answered Feb 21 '14 at 15:06

3

You can find a good explanation in the django LiveServerTestCase

class LiveServerTestCase(TransactionTestCase):
    """
    ...
    Note that it inherits from TransactionTestCase instead of TestCase because
    the threads do not share the same transactions (unless if using in-memory
    sqlite) and each thread needs to commit all their transactions so that the
    other thread can see the changes.
    """

answered Feb 21 '14 at 15:06

erthalion

3,094
2
21
28

This is definitely a good point. However, it says: "each thread needs to commit all their transactions so that the other thread can see the changes". But, in this case, I'm not even able to see the initial data upon loading the fixtures. – Gaurav Toshniwal Feb 21 '14 at 15:19
I think I got it. Basically, in case of a TestCase, the fixture is being loaded in that particular transaction and hence according to this part of the documentation, the other thread cannot see this, because the fixture altogether is not yet committed. Sounds legit? – Gaurav Toshniwal Feb 22 '14 at 06:30
Yes, of course - because the fixtures loading doesn't different from the usual data saving in the test body. – erthalion Feb 23 '14 at 02:54

Django unittesting: model.all() queryset is empty in a method running in separate thread

1 Answers1