0

I'm using "DATABASE_ROUTERS" and just added migrations to new database. I have 153 migrations on the "default" database and it seems that all of those have to be run on the new database even though they don't apply. My db router's allow_migrate returns False on every migration except for the one "initial" one related to the new database.

I faked that one initial migration and then ran manage.py migrate --database new_database and was surprised that after 45 minutes I had to kill the process when it had used up all the memory and all of the swap space!

I then tried again but this time with manage.py migrate --database new_database --fake and it seemed to make no difference. My memory and swap usage went through the roof and I had to again kill the process. All this command should be doing is marking all migrations as completed in the "django_migrations" table. What is really happening to cause so many resources to be used? Am I doing something wrong?

What is the best way around this issue? Should I just manually create the "django_migrations" table and then populate it myself?

Tim Tisdall
  • 9,914
  • 3
  • 52
  • 82
  • Is this app deployed many places? If there's no deployment that requires all those old migrations, you might as well squash the existing migrations or delete them and build shiny new minimal migrations. https://stackoverflow.com/questions/40028586/how-to-squash-recent-django-migrations – Håken Lid Nov 03 '17 at 14:19
  • @HåkenLid - on the "default" database there's no issue when there's just a few new migrations to run. The issue seems to be when running it all from scratch. Regardless, a "--fake" shouldn't use all system resources and take over 45 min. Also, once it finally does completely run, it's okay after that for just a few new migrations. – Tim Tisdall Nov 03 '17 at 14:27
  • I agree it's strange that `--fake` takes such a long time. Even more reason to just trash those migrations and start over. – Håken Lid Nov 03 '17 at 14:32
  • 1
    For context, we run about 250 migrations on our Django project when deploying (including creating objects in migrations). It takes about 2 minutes on my machine (with data), and about 45 seconds on the server. Around 20 seconds on an empty db. Dumping and rebuilding that database takes about 6 minutes. – Withnail Nov 03 '17 at 15:36
  • @Withnail - I think this may be an issue specific to Django 1.8 (there were issues with earlier subversions). Which version of Django are you using? – Tim Tisdall Nov 03 '17 at 15:56
  • Ah, interesting. (Just for clarity - i regard these migration speeds as non problematic!) We're on 1.10. – Withnail Nov 03 '17 at 16:13
  • @Withnail - My issue isn't so much the time as the resources taken... I can let it run for an hour, but if it uses all the CPU, RAM, and swap to do it, then the server won't be working for anything else. – Tim Tisdall Nov 03 '17 at 18:18

1 Answers1

0

In case other users come to this, here's what I did to solve it (but it's not the most ideal). I essentially faked the django_migrations table since none of the the migrations actually needed to be made in the new database.

First I looked in the "default" database's django_migrations table to see what the first migration was and then ran that via the command line to create the initial table in the new database. For me that was this:

python manage.py migrate --database new_database --fake contenttypes 0001_initial

After I had the first initial value and the table created I essentially copied over the rest using an SQL statement. I made sure to retain the order from the "default" database in case that had any importance. Statement looked like this:

INSERT INTO django_migrations (app, name, applied) VALUES 
('auth', '0001_initial', NOW()),
('users', '0001_initial', NOW()),
  ...

Then just to make sure everything was good I ran:

python manage.py migrate --database new_database
python manage.py migrate
Tim Tisdall
  • 9,914
  • 3
  • 52
  • 82