135

My Django unit tests take a long time to run, so I'm looking for ways to speed that up. I'm considering installing an SSD, but I know that has its downsides too. Of course, there are things I could do with my code, but I'm looking for a structural fix. Even running a single test is slow since the database needs to be rebuilt / south migrated every time. So here's my idea...

Since I know the test database will always be quite small, why can't I just configure the system to always keep the entire test database in RAM? Never touch the disk at all. How do I configure this in Django? I'd prefer to keep using MySQL since that's what I use in production, but if SQLite 3 or something else makes this easy, I'd go that way.

Does SQLite or MySQL have an option to run entirely in memory? It should be possible to configure a RAM disk and then configure the test database to store its data there, but I'm not sure how to tell Django / MySQL to use a different data directory for a certain database, especially since it keeps getting erased and recreated each run. (I'm on a Mac FWIW.)

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Leopd
  • 41,333
  • 31
  • 129
  • 167

8 Answers8

177

If you set your database engine to sqlite3 when you run your tests, Django will use a in-memory database.

I'm using code like this in my settings.py to set the engine to sqlite when running my tests:

if 'test' in sys.argv:
    DATABASE_ENGINE = 'sqlite3'

Or in Django 1.2:

if 'test' in sys.argv:
    DATABASES['default'] = {'ENGINE': 'sqlite3'}

And finally in Django 1.3 and 1.4:

if 'test' in sys.argv:
    DATABASES['default'] = {'ENGINE': 'django.db.backends.sqlite3'}

(The full path to the backend isn't strictly necessary with Django 1.3, but makes the setting forward compatible.)

You can also add the following line, in case you are having problems with South migrations:

    SOUTH_TESTS_MIGRATE = False
Endre Both
  • 5,540
  • 1
  • 26
  • 31
Etienne
  • 12,440
  • 5
  • 44
  • 50
  • Do you put that in your settings.py? – Leopd Jun 23 '10 at 20:34
  • 9
    Yes, exactly. I should have put that in my answer! Combine that with SOUTH_TESTS_MIGRATE = False and your tests should be a lot faster. – Etienne Jun 24 '10 at 19:19
  • Awesome! With these two changes the overhead for running a single fast test in my system has gone from 12 seconds down to <100ms. – Leopd Jun 24 '10 at 19:58
  • 7
    this *is* awesome. on newer django setups use this line: 'ENGINE': 'sqlite3' if 'test' in sys.argv else 'django.db.backends.mysql', – mjallday Jan 19 '11 at 06:26
  • -1, because testing on different database than the production one is pointless, especially if it's as limited database as SQLite. – Tomasz Zieliński Mar 06 '11 at 20:45
  • 3
    @Tomasz Zielinski - Hmm, it depends what you are testing. But I totally agree that, at the end and from time to time, you need to run the tests with your real database (Postgres, MySQL, Oracle...). But running your tests in-memory with sqlite can save you a lot of time. – Etienne Mar 07 '11 at 03:02
  • @Etienne: Yes, it can save time, but you can also run tests using your primary database copied to ramdisk, it's very easy with MySQL and then it's not much slower than SQLite. SQLite if *very* different from MySQL, so even if all tests pass on it, it doesn't mean at all that they will pass with MySQL. – Tomasz Zieliński Mar 07 '11 at 08:42
  • With the new Django syntax, that should be: if 'test' in sys.argv: DATABASES['default']['ENGINE'] = 'django.db.backends.sqlite3' Great solution! – Pablo Alba Jun 03 '11 at 10:52
  • 2
    Isn't Django supposed to be database-agnostic? It *should* function pretty much the same with every DB, right? – Tuukka Mustonen Jun 10 '11 at 06:25
  • Also, does this really turn on in-memory engine? If it does that automatically, what would you need to do to force sqlite to use flat-file engine instead? – Tuukka Mustonen Jun 10 '11 at 06:36
  • 1
    If I'm right, if you specified a TEST_NAME in the settings, SQLite will use a flat-file. For your first question, has you write, it should function pretty much the same, but it will not necessarily function exactly the same. That's why you really need to run your tests, at least one time, on the DB engine you're using in production before pushing your code to production. – Etienne Jun 10 '11 at 13:35
  • 2
    @Etienne: Ah, you are correct. When running Django (`runserver`), Django will make sqlite use flat-file DB but when running tests and without `TEST_NAME` it will use in-memory DB instead (not flat-file, as I presumed). Thanks for pointing that out. You might want to add note to your answer about that (leaving `TEST_NAME` as `None` or specifying as `:memory:` results in in-memory DB, anything else results in flat-file DB). Otherwise, someone might use sqlite engine but have `TEST_NAME` set to something, and wonder why flat-file is used instead. – Tuukka Mustonen Jun 18 '11 at 13:20
  • 3
    I reverse -1 to +1: as I see it now, it's much faster to use sqlite for quick runs and switch to MySQL for e.g. final daily tests. (Note that I had to do a dummy edit to unlock voting) – Tomasz Zieliński Jun 25 '11 at 21:36
  • 1
    Please note that in-memory SQLite3 DBs do not support access from multiple threads, at least in my setup! I just ran into this issue and had to switch back to MySQL... :-( – JohnJ Feb 09 '12 at 21:36
  • With Django 1.4 you need to specify the full backend name: 'ENGINE': 'django.db.backends.sqlite3'. – AJJ May 05 '12 at 16:36
  • Quick note; you can use SQLite for running tests locally and then use your production DB engine (MySQL, PostgreSQL, etc...) for testing if you have a continuous integration server. – Dana Woodman May 31 '12 at 21:17
  • 2
    This doesn't seem to work on Django>=1.5. It now complains about the NAME field being missing. – Cerin May 01 '14 at 14:33
  • 14
    Caution with this `"test" in sys.argv`; it may trigger when you don't want it to, e.g. `manage.py collectstatic -i test`. `sys.argv[1] == "test"` is a more precise condition that should not have that problem. – keturn Jan 08 '15 at 23:23
  • This approach seems like it could easily lead to some "gotcha!" situations. It might get the job done, but having a separate settings file is more robust and explicit about what's happening. See Anurag Uniyual's answer below: http://stackoverflow.com/a/11018426/1752050 – doctaphred Jul 19 '16 at 18:06
  • @doctaphred: I totally agree with you. You know, this answer is now 6 years old! – Etienne Jul 19 '16 at 22:09
86

I usually create a separate settings file for tests and use it in test command e.g.

python manage.py test --settings=mysite.test_settings myapp

It has two benefits:

  1. You don't have to check for test or any such magic word in sys.argv, test_settings.py can simply be 

    from settings import *

# make tests faster
SOUTH_TESTS_MIGRATE = False
DATABASES['default'] = {'ENGINE': 'django.db.backends.sqlite3'}

    Or you can further tweak it for your needs, cleanly separating test settings from production settings.

  2. Another benefit is that you can run test with production database engine instead of sqlite3 avoiding subtle bugs, so while developing use

    python manage.py test --settings=mysite.test_settings myapp

    and before committing code run once

    python manage.py test myapp

    just to be sure that all test are really passing.

Anurag Uniyal
  • 85,954
  • 40
  • 175
  • 219
  • 2
    I like this approach. I have a bunch of different settings files and use them for different server environments but I had not thought about using this method to choose a different test database. Thanks for the idea. – Mentakatz Jul 19 '13 at 21:02
  • Hi Anurag, I tried this but my other databases mentioned in the settings are also executed. I am not able to figure out the exact reason. – Bhupesh Pant Jun 11 '14 at 13:20
  • Nice answer. I wonder how to specify a settings file when running tests through coverage. – Wtower Mar 10 '15 at 13:32
  • It's a good approach, but not DRY. Django already knows that you're running tests. If you could 'hook into' this knowledge somehow, you'd be set. Unfortunately, I believe that requires extending the management command. It would probably make sense to make this generic in the core of the framework, by, for example having a setting MANAGEMENT_COMMAND set to the current command whenever manage.py is called, or something to that effect. – DylanYoung Dec 13 '17 at 18:40
  • 2
    @DylanYoung you can make it dry by including main settings into test_settings and just overriding things you want for test. – Anurag Uniyal Dec 30 '17 at 04:52
  • You misunderstand my point. It's not DRY because of the argv test: Django already knows which command you're running. – DylanYoung Dec 31 '17 at 21:35
  • @DylanYoung that is easy, you can write a custom command, you can even override the 'test' command https://docs.djangoproject.com/en/2.0/howto/custom-management-commands/ but that doesn't add much , at-least in my case I wanted to optionally pass different settings not always. – Anurag Uniyal Jan 11 '18 at 07:33
  • Yes I know. That's what I said: "Unfortunately, I believe that requires extending the management command.". A shell alias would be easier, but less portable :) – DylanYoung Jan 11 '18 at 18:51
  • `before committing code run once "python manage.py test myapp"`. I think this should be automated such as using CI or git commit hook. Because people are "lazy", they are not willing to run slow tests while there are quick tests or they will always forget to change the command of different settings. – Cloud Jul 18 '18 at 02:24
21

MySQL supports a storage engine called "MEMORY", which you can configure in your database config (settings.py) as such:

    'USER': 'root',                      # Not used with sqlite3.
    'PASSWORD': '',                  # Not used with sqlite3.
    'OPTIONS': {
        "init_command": "SET storage_engine=MEMORY",
    }

Note that the MEMORY storage engine doesn't support blob / text columns, so if you're using django.db.models.TextField this won't work for you.

muudscope
  • 6,780
  • 4
  • 21
  • 20
  • 6
    +1 for mentioning lack of support for blob/text columns. It also doesn't seem to support transactions (http://dev.mysql.com/doc/refman/5.6/en/memory-storage-engine.html). – Tuukka Mustonen Jun 10 '11 at 06:27
  • If you really want in-memory tests, you're probably better off going with sqlite which at least supports transactions. – atomic77 Jan 23 '18 at 20:54
14

I can't answer your main question, but there are a couple of things that you can do to speed things up.

Firstly, make sure that your MySQL database is set up to use InnoDB. Then it can use transactions to rollback the state of the db before each test, which in my experience has led to a massive speed-up. You can pass a database init command in your settings.py (Django 1.2 syntax):

DATABASES = {
    'default': {
            'ENGINE':'django.db.backends.mysql',
            'HOST':'localhost',
            'NAME':'mydb',
            'USER':'whoever',
            'PASSWORD':'whatever',
            'OPTIONS':{"init_command": "SET storage_engine=INNODB" } 
        }
    }

Secondly, you don't need to run the South migrations each time. Set SOUTH_TESTS_MIGRATE = False in your settings.py and the database will be created with plain syncdb, which will be much quicker than running through all the historic migrations.

Daniel Roseman
  • 588,541
  • 66
  • 880
  • 895
  • Great tip! It reduced my tests from `369 tests in 498.704s` to `369 tests in 41.334s `. This is more than 10 times faster! – Gabi Purcaru Jan 04 '12 at 10:17
  • Is there an equivalent switch in settings.py for migrations in Django 1.7+ ? – Edward Newell Mar 10 '15 at 20:25
  • @EdwardNewell Not exactly. But you can use `--keep` to persist the database and not require your complete set of migrations to be reapplied on every test run. New migrations will still run. If you're switching between branches frequently, it's easy to get into an inconsistent state though (you can revert new migrations before you switch by changing the database to the test database and running `migrate`, but it's a bit of a pain). – DylanYoung Dec 13 '17 at 18:44
10

You can do double tweaking:

  • use transactional tables: initial fixtures state will be set using database rollback after every TestCase.
  • put your database data dir on ramdisk: you will gain much as far as database creation is concerned and also running test will be faster.

I'm using both tricks and I'm quite happy.

How to set up it for MySQL on Ubuntu:

$ sudo service mysql stop
$ sudo cp -pRL /var/lib/mysql /dev/shm/mysql

$ vim /etc/mysql/my.cnf
# datadir = /dev/shm/mysql
$ sudo service mysql start

Beware, it's just for testing, after reboot your database from memory is lost!

Potr Czachur
  • 2,066
  • 1
  • 14
  • 6
  • thanks! works for me. I can't use sqlite, because I'm using features specific to mysql (full-text indexes). For ubuntu users, you'll have to edit your apparmor config to allow mysqld access to /dev/shm/mysql – Ivan Virabyan May 04 '11 at 09:47
  • Cheers for the heads up Ivan and Potr. Disabled the AppArmor mysql profile for now, but found a guide for customising the relevant local profile: https://blogs.oracle.com/jsmyth/entry/apparmor_and_mysql – trojjer Aug 01 '13 at 10:57
  • Hmm. I've tried customising the local profile to give mysqld access to the /dev/shm/mysql path and its contents, but the service can only start in 'complain' mode (aa-complain command) and not 'enforce', for some reason... A question for another forum! What I can't understand is how there are no 'complaints' at all when it does work, implying that mysqld isn't violating the profile... – trojjer Aug 01 '13 at 11:03
4

Another approach: have another instance of MySQL running in a tempfs that uses a RAM Disk. Instructions in this blog post: Speeding up MySQL for testing in Django.

Advantages:

  • You use the exactly same database that your production server uses
  • no need to change your default mysql configuration
neves
  • 33,186
  • 27
  • 159
  • 192
2

Extending on Anurag's answer I simplified the process by creating the same test_settings and adding the following to manage.py

if len(sys.argv) > 1 and sys.argv[1] == "test":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.test_settings")
else:
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")

seems cleaner since sys is already imported and manage.py is only used via command line, so no need to clutter up settings

Alvin
  • 2,533
  • 33
  • 45
  • How does this let you ever run against your production db? – boatcoder Oct 25 '12 at 22:19
  • would need to be extended to so -- or you could set the settings file manually via the command line -- question of which you test on more often – Alvin Oct 26 '12 at 06:39
  • I like this approach, but to be clear, will passing --settings=mysite.settings use settings and not test_settings? – yellottyellott Apr 03 '13 at 04:54
  • 2
    Caution with this `"test" in sys.argv`; it may trigger when you don't want it to, e.g. `manage.py collectstatic -i test`. `sys.argv[1] == "test"` is a more precise condition that should not have that problem. – keturn Jan 08 '15 at 23:23
  • 2
    @keturn this way it generates an exception when running `./manage.py` without arguments (eg to see which plugins are available, same as `--help`) – Antony Hatchkins Jul 25 '15 at 09:05
  • 1
    @AntonyHatchkins That's trivial to resolve: `len(sys.argv) > 1 and sys.argv[1] == "test"` – DylanYoung Dec 13 '17 at 15:32
  • 1
    @DylanYoung Yes, that's exactly what I wanted Alvin to add to his solution but he isn't particularly interested in improving it. Anyway it looks more like a quick hack than the legit solution. – Antony Hatchkins Dec 13 '17 at 15:52
  • @AntonyHatchkins It is a bit hacky, but as long as you have a standard testing process (that involves `test` as the first arg) it should work fine. A clean solution implementing the same general principle, would be to extend the Django test management command and set a variable there. – DylanYoung Dec 13 '17 at 18:34
  • 1
    haven't looked at this answer in a while, I updated the snippet to reflect @DylanYoung's improvement – Alvin Dec 13 '17 at 19:09
  • 1
    @DylanYoung Actually it could be written even more hacky (although less readable): `if sys.argv[1:2] == ['test']:` but as a reader I'd prefer the more verbose variant :) – Antony Hatchkins Dec 14 '17 at 06:13
-1

Use below in your setting.py

DATABASES['default']['ENGINE'] = 'django.db.backends.sqlite3'
Ehsan
  • 3,711
  • 27
  • 30