22

I have a app that is deployed to Heroku, and I'd like to be able to run the test suite post-deployment on the target environment. I am using the Heroku Postgres add-on, which means that I have access to a single database only. I have no rights to create new databases, which in turn means that the standard Django test command fails, as it can't create the test_* database.

$ heroku run python manage.py test
Running `python manage.py test` attached to terminal... up, run.9362
Creating test database for alias 'default'...
Got an error creating the test database: permission denied to create database

Is there any way around this?

Hugo Rodger-Brown
  • 11,054
  • 11
  • 52
  • 78

2 Answers2

14

Turns out I was in the wrong. I was not testing what I thought was being tested... Since Heroku's Routing Mesh was sending requests to different servers, the LiveServerTestCase was starting a web server on one machine and Selenium was connecting to other machines altogether.

By updating the Heroku Procfile to:

web: python src/manage.py test --liveserver=0.0.0.0:$PORT

overriding the DATABASES setting to point to the test database, and customizing the test suite runner linked to below (the same idea still holds: override setup_databases so that it only drops/re-creates tables, not the entire database), I was able to run remote tests. But this is even more hacky/painful/inelegant. Still looking for something better! Sorry about the confusion.

(updated answer below)

Here are the steps that worked for me:

  • Create an additional, free Postgres database using the Heroku toolbelt

heroku addons:add heroku-postgresql:dev

  • Use the HerokuTestSuiteRunner class which you'll find here.

This custom test runner requires that you define a TEST_DATABASES setting which follows the typical DATABASES format. For instance:

TEST_DATABASES = {
    'default': dj_database_url.config(env='TEST_DATABASE_URL')
}

Then, have the TEST_RUNNER setting be a Python path to wherever HerokuTestSuiteRunner can be found.

You should now be able to run Django tests on Heroku using the given database. This is very much a quick hack... Let me know how it could be improved / made less hackish. Enjoy!


(original answer below)

A few relevant solutions have been discussed here. As you can read in the Django docs, "[w]hen using the SQLite database engine, the tests will by default use an in-memory database".

Although this doesn't thoroughly test the database engine you're using on Heroku (I'm still on the lookout for a solution that does that), setting the database engine to SQLite will at least allow you to run your tests.

See the above-linked StackOverflow question for some pointers. There are at least two ways out: testing if 'test' in sys.argv before forcing SQLite as the database engine, or having a dedicated settings file used in testing, which you can then pass to django manage.py test using the --settings option.

Community
  • 1
  • 1
Greg Sadetsky
  • 4,863
  • 1
  • 38
  • 48
  • 2
    Thanks Greg - using SQLite isn't really an option - we've had some horrendous problems caused by tests passing with SQLite (no constraints / consistency checks) that have subsequently blown up on live. Current solution is to replicate Heroku as close as poss in a VM (using Vagrant) and run them there. Works pretty well, and no disasters so far. – Hugo Rodger-Brown Jan 18 '13 at 00:00
  • 1
    Thanks Hugo, that's good to know. I won't be using SQLite for testing then either (as I use Postgres in production). Why not add an additional, free 'heroku-postgresql:dev' database to your environment on which tests could be run? From what I can gather, it is possible to have multiple instances of the same add-on. – Greg Sadetsky Jan 20 '13 at 19:14
  • Damnation - that doesn't work. Django drops the entire database (rather than the tables / indices therein) and the Heroku postgres addon doesn't give the user create_db privileges. Good idea though. – Hugo Rodger-Brown Jan 21 '13 at 17:00
  • 1
    Thanks for testing that! But let's not give up! :-) Looking at the [django.test](https://github.com/django/django/blob/master/django/test/simple.py) source code, it seems that inheriting from DjangoTestSuiteRunner and overriding both `setup_databases` and `teardown_databases` would do the trick, no? Can't test this tonight, but I am looking forward to soon. Feel free to beat me to the punch! – Greg Sadetsky Jan 23 '13 at 01:51
  • 1
    please see my edit to the answer with a more thorough, Postgres-compatible solution. – Greg Sadetsky Feb 23 '13 at 16:53
  • 1
    There's a big helpful conversation on this topic at https://gist.github.com/cordery/d52d9ba44541fabaf4b012f4e62d675b – Dave Aaron Smith Feb 26 '18 at 22:15
3

Starting with version 1.8, Django now has an option called keepdb, which allows for the same database to be reused during tests.

The --keepdb option can be used to preserve the test database between test runs.

This has the advantage of skipping both the create and destroy actions which can greatly decrease the time to run tests, especially those in a large test suite.

If the test database does not exist, it will be created on the first run and then preserved for each subsequent run.

Any unapplied migrations will also be applied to the test database before running the test suite.

Since it also allows for the test database to exist prior to running tests, you can simply add a new Postgres Heroku instance to your dyno and configure the tests to use that particular database.

Bonus : you can also use the failfast option, which exits as soon as your first test crashes, so that you don't have to wait for all tests to complete.

However, if you are deploying things to Heroku and you are using Heroku Pipelines, an even better option is available : Heroku CI.

Community
  • 1
  • 1
AdelaN
  • 3,366
  • 2
  • 25
  • 45