Parrallel south migration in django causes errors

Question

I ran a code update that points at two front end servers (Amazon Web Service Instances).

A south migration was included as part of the update.Since the migration the live site appears to flit between the current code revision , and the previous revision, at will.

Since discovering this, A previous developer (who has left the company before I turned up), said, & I quote:

"never run migrations in parallel. Running migrations twice causes duplication of new objects >and other errors!"

My code changes did not involve any models.py changes ; the migrate commands were just part of the fabric update script. Also no errors were thrown during the migrations, they seemingly ran as normal.

I have database backups, so I can roll back the database as a last resort. Is there any other way to sort the issue without doing this?

Thanks for reading

edit: I should add, I pushed the same code to a staging server and it worked fine, so the issue isnt the code

Answer 1

Firstly, if one migration tried to add a table or column that another migration already added, while the other migration was running, the first migration would fail, rolling back the transaction (i.e., not changing anything). So, you shouldn't really be able to run into a problem like your peer described.

However, if you really do somehow have that problem (partially applied migrations, etc.), the following are a couple of options:

Option 1

1. Reverse the migration, if it's safely reversible, then run it again (only once this time :)
2. Remove migrations from your deployment script

Option 2

1. Open a Postgres (or other DBMS) shell, and check for the existence of columns or tables that you fear might have been created, then remove them, if needed, the goal being to reverse all of the (partial) effects of the migration having been run
2. Open a Django shell and import MigrationHistory from South's models module
3. Find the MigrationHistory objects that pertain to the migrations that were run, and delete them
4. Run the migration again
5. Remove migrations from your deployment script

Note, the final step of each option is "remove migrations from your deployment script". This is because you shouldn't have a database altering statement running while your deployment goes. Even though it feels all wrong, and old-school, you should really run your migrations after the deployment, stopping your service entirely during deployment in cases where there could be problems if new code was deployed with old database tables, and leaving your app running when the migrations are less prone to problems arising from the aforementioned.

Answer 2

Turns out the problem was that the git fetch on one of the front servers didnt take, which is what was causing the problem..it had nothing to do with running migrations in parallel (though I shouldnt have done that anyway)