Django 1.10 seed database without using fixtures

Question

So I've looked at the documentation, as well as this SO question, and the django-seed package, but none of these seem to fit what I'm trying to do.

Basically, I want to programmatically seed my Games model from an external API, but all the information I can find seems to be reliant on generating a fixture first, which seems like an unnecessary step.

For example, in Ruby/Rails you can write directly to seed.rb and seed the database in anyway that's desired.

If a similar functionality available in Django, or do I need to generate the fixture first from the API, and then import it?

Django's `manage.py loaddata` just parses your text file and inserts it into your database. You can do that programatically with an initial migration, like it recommends at the bottom of your [linked documentation](https://docs.djangoproject.com/en/1.11/howto/initial-data/#providing-initial-data-with-migrations). — Blender, Nov 14 '17 at 22:58
Yes, but I'm trying to avoid making the text file. Seems unnecessary. — Darkstarone, Nov 14 '17 at 23:02
Migrations don't need to contain any hard-coded initial data, they're just Python scripts. You can just use a `RunPython` migration and create a function that streams the data from your API and inserts it into your database. It will be run automatically when the initial migrations are run and will be resistant to corrupting your database state because migrations utilize transactions automatically. — Blender, Nov 14 '17 at 23:08
Ah with you now. I didn't realize Django's docs suggested creating data migrations. Still getting used to the changing terminology. Thanks for taking the time to explain. — Darkstarone, Nov 14 '17 at 23:14

score 5 · Accepted Answer · answered Nov 14 '17 at 23:22

You can use a data migration for this. First create an empty migration for your app:

$ python manage.py makemigrations yourappname --empty

In your empty migration, create a function to load your data and add a migrations.RunPython operation. Here's a modified version of the one from the Django documentation on migrations:

from __future__ import unicode_literals
from django.db import migrations

def stream_from_api():
    ...

def load_data(apps, schema_editor):
    # We can't import the Person model directly as it may be a newer
    # version than this migration expects. We use the historical version.
    Person = apps.get_model('yourappname', 'Person')

    for item in stream_from_api():
        person = Person(first=item['first'], last=item['last'], age=item['age'])
        person.save()

class Migration(migrations.Migration):
    dependencies = [('yourappname', '0009_something')]
    operations = [migrations.RunPython(load_data)]

If you have a lot of simple data, you might benefit from the bulk-creation methods:

from __future__ import unicode_literals
from django.db import migrations

def stream_from_api():
    ...

def load_data(apps, schema_editor):
    # We can't import the Person model directly as it may be a newer
    # version than this migration expects. We use the historical version.
    Person = apps.get_model('yourappname', 'Person')

    def stream_people():
        for item in stream_from_api():
            yield Person(first=item['first'], last=item['last'], age=item['age'])

    # Adjust (or remove) the batch size depending on your needs.
    # You won't be able to use this method if your objects depend on one-another
    Person.objects.bulk_create(stream_people(), batch_size=10000)

class Migration(migrations.Migration):
    dependencies = [('yourappname', '0009_something')]
    operations = [migrations.RunPython(load_data)]

Migrations have the added benefit of being automatically enclosed in a transaction, so you can stop the migration at any time and it won't leave your database in an inconsistent state.

In your `load_data` function, anyway to use `apps.get_model()` to load the base User model? I know there is, just not sure how to do it. — Tunn, Dec 21 '17 at 16:54

score 1 · Answer 2 · answered Nov 14 '17 at 22:57

1

Would it work for you to write some class method on the Games model that creates the data? Presumably this method query the external API, package up the Games() objects as a list called games and then use a Games.objects.bulk_create(games) to insert it into the database.

answered Nov 14 '17 at 22:57

VMatić

996
2
10
18

This seems fairly ideal. Thanks. – Darkstarone Nov 14 '17 at 23:01

Django 1.10 seed database without using fixtures

2 Answers2