Django - in memory sqlite in production

Question

I have a small (10MB), read-only, sqlite3 DB that I use in production.

I want to speed up my website so I'm trying to load the entire DB from disk to memory on every Django startup.

Is there a similar solution for Django?

Probably not if you want to use Django's ORM, as that requires generating the schema beforehand. For a small database, most of the data should be cached anyway, so explicitly loading it in RAM is unlikely to help. — Colonel Thirty Two, Nov 25 '15 at 15:45
what do you mean by "generating the schema beforehand"? it's True that it's cached but I think the default cache size is only 2MB and also the DB might get larger to 50MB so I would still prefer to explicitly load it to memory — Dan Bolofe, Nov 25 '15 at 21:39
I mean the `CREATE TABLE` statements. Django doesn't run those at startup; it requires that you run `manage.py setup` or `manage.py migrate` first. The cache size is tunable via the [`cache_size` pragma](http://sqlite.org/pragma.html#pragma_cache_size) (which you will probably need to alter using raw SQL statements). — Colonel Thirty Two, Nov 25 '15 at 21:50
But I'm trying to load an existing database to memory using iterdump() and then executescript() like in this answer http://stackoverflow.com/a/10856450/3327587 so I don't really need to worry about "create table" as it's covered by iterdump(). My problem is that after creating the memory database with all the data I don't know how to tell django to use this memory db — Dan Bolofe, Nov 27 '15 at 11:31

score 40 · Accepted Answer · edited May 23 '17 at 12:26

40

Configure memory database:

DATABASES = {
  'default': {
    'ENGINE': 'django.db.backends.sqlite3',
    'NAME': ':memory:',
  }
}

and put the code you've linked to as a startup script (please refer to Execute code when Django starts ONCE only?).

edited May 23 '17 at 12:26

Community

answered Nov 25 '15 at 15:46

Adrian

1

But django will create a new database in memory, how do i tell django to use the existing memory DB that I loaded? I need to point django to a connection or a StringIO object or something.. – Dan Bolofe Nov 25 '15 at 21:34
@DanBolofe You cannot load a byte array as a database; SQLite simply doesn't support doing that. – Colonel Thirty Two Nov 25 '15 at 21:52
You can if you dump everything as sql statements and then execute them - see: http://stackoverflow.com/a/10856450/3327587 – Dan Bolofe Nov 27 '15 at 11:23
@DanBolofe Then run the SQL script, using the instructions on how to run code when Django starts that Adrian linked to. – Colonel Thirty Two Nov 27 '15 at 13:25
I guess that's really the best option, to dump sql and then execute raw sql using cursor.execute(), thanks – Dan Bolofe Nov 28 '15 at 14:14

1 Answers1