Django unit test query against production database

Question

I am trying to write some unit tests for an asynchronous method in my Django app. Essentially, when a user does a certain POST, this method is kicked off so the application doesn't hang. What I want to test is the ability to cancel this request as it is running (in this case via celery). The only problem with doing this is because Celery is running independent of the web application, it pushes the results to the real database, not the test database that is created by Django Unit Tests. So what I'm wondering is how would I be able to do something like

Results.objects.get(id=some_id)

and tell it to point to the actual database. I've already tried the following:

Results.objects.using('default').get(id=some_id)

Thinking that maybe this would solve the problem, but it didn't find the result (again because it was pushing the information to the actual database, not the test DB). I did some searching around and found this link: How do I run a unit test against the production database? but the answers on here say things like "You shouldn't test against production." I completely understand this is not good practice but I need to hit the live database. Everything is being run on a VM so it's all a test database anyways. Does anyone know of a way to hit the real database?

Thanks,

How are you starting your celery workers ?? – dm03514 Aug 13 '16 at 16:49 — dm03514, Aug 13 '16 at 16:49

score 1 · Answer 1 · answered Jun 17 '19 at 21:01

There is one thing we can do, a dirty trick!

Run your celery worker and scheduler on a setting which uses say 'new_one' database.

You should have a settings.py like this -

DATABASES = {
   'default': {
       'TEST': {
           'NAME': 'new_one',
       },
   } 
} 
db_from_env = dj_database_url.config(default=config('DATABASE_URL'))
DATABASES['default'].update(db_from_env)

After that run your django server on another database say
'another_new_one'.
Run your test cases, they will pass. Why? Test cases are using 'new_one' database, celery also using 'new_one' database, your django server using 'another_new_one'. Cheers!

I have tried giving same names to test and production database, but error comes when running tests 'database already in use by someone else', that someone else is django server. So need two different databases, 1- new_one 2-another_new_one

  Note- I have used .env file to use config(), where I have my production database name host user password etc. It can change according to what you use. Just use two different databases and do this dirty trick, it will work. As it worked for me.

score 0 · Answer 2 · answered Aug 13 '16 at 16:48

I'm not sure if testing against your prod is a better solution than running your celery workers in dev mode against your test database. How are you starting your celery workers ?? I imagine you're starting your celery workers external of the test using your prod/real settings, then you're running your tests using the test settings. For an integration test, it would probably be more repeatable/maintainable if you had an integration environment.

a common test strategy for this would be:

unit test your view to make sure that it calls the correct celery task with the correct arguments, you could stub out the celery task dependency, or you could run the celery task syncronously using CELERY_ALWAYS_EAGER=True, and an in memory Queue.
if you stub out the celery dependency, then there should be a separate unit test exercising its logic

Your original scenario is not super well suited for django unit test framework because of the inter process dependency of the celery broker. An integration test between your app and celery would probably be valuable, but for it to be maintainable, repeatable, and reliable celery broker process startup, cleanup, etc should probably be managed in an automated way.

Ideally I would be able to run the celery task synchronously yes. Unfortunately, the way the program is coded, it has to run asynchronously in the unit test because it is constantly checking the status of a certain object. If the value of the a string field of this object is set to "CANCEL" the task is canceled mid execution. Obviously if I'm running this synchronously I would have the problem of not being able to test this "CANCEL" case. — Michael Platt, Aug 15 '16 at 14:22

score 0 · Answer 3 · edited May 23 '17 at 12:18

If you really want to run dedicated celery worker process from unit test, then you need to start celery worker from setUp() (or whatever you want method) and copy django.conf.settings.DATABASE from test process to celery worker process.

For starting point see my answer on very similar question https://stackoverflow.com/a/42107423/590233 (there is code that starts celery worker)

And here is sample code that configures django and DATABASE settings within single module https://gist.github.com/Sovetnikov/369a8d05ba2b6482fa20769bc498f122

Just combine two techniques and you will get what you want.

Django unit test query against production database

3 Answers3