1

I'm building my first migration system ever with Umzug and Sequelize for a node app which uses sequelize and express on the backend.

My issue is that as the back end models change with commits over time (models being deleted, some changing, some being added) this breaks old migrations that happen to use the models from sequelize. Example:

Lets say I have a migration #1 that deals with "UserStats". Five releases later model UserStats needs to be removed from the app, so the model is deleted and a new migration is made to drop the table.

Now attempting to startup a new dev environment breaks because when the new server tries to run through all the old migrations it tries to find the model UserStats for the first migration, but the model no longer exists.

So the root problem is that versions of models get out of sync with the app migration state. Each migration requires the sequelize models to look like they did when that migration was initially created. What is the best way to deal with this?

Henry
  • 491
  • 4
  • 13

3 Answers3

2

There's two ideas worth knowing about here. Both are inspired from my background in Ruby on Rails, so I've included a bit about how Rails does it for context:

  1. In Rails, when you set up a new database, it doesn't expect to run every migration ever, to get the database right. Rails also maintains a single file called schema.rb which holds the current, full, up-to-date state of your database (including a list of the names of the migrations that have run to get it up to that point). So when you create a new database, it just reads whatever's in the schema.rb file, and creates those tables.

Then, rails will run FUTURE migrations on that database, moving forward. But the ones that had run prior to the latest schema.rb being created - well, you don't need them at all. In fact, once they've run everywhere they need to, you can delete them if you want.

Now, sequelize doesn't do this (though I wish it did!). What you can do, though, is every so often, dump your entire database structure as SQL. Save it as eg. 20170427051240-initial-structure.sql, in your migrations file, timestamped to be a second after the latest migration that it included. So, now you have something kinda like Rail's scehma.rb.

The next step is: edit the migration that ran directly prior to that timestamp, so that all it does is imports that entire database structure. Something like this:

'use strict';

module.exports = {
  up: function(queryInterface, Sequelize) {
    if (!['test', 'ci'].includes(process.env.NODE_ENV)) {
      // In all other environments, the database will already have been set up, so
      // we don't need to import the full structure
      return;
    }
    return queryInterface.sequelize.query(`
      BIG FAT STRING OF SQL GOES HERE.
      Note: in my app I've actually got it done table by table,
      in separate chained promises, but don't remember why.
    `)
  }
}

OK. So now you can delete all migrations prior to that one - because that one takes care of everything that would have run prior, anyway.

There's a few considerations, eg. you might want to save 20170427051240-initial-structure.sql, then wait a month or two, allow another 15 migrations to accumulate, and then do the steps above, so that setting up a new database will import the initial structure as it was a month ago, in the first migration, and then run the most recent 15 migrations, on top of that. That means you always keep a record of the last several migrations, in case you need to roll them back or something.

  1. Another common practice in the Rails world, which more directly applies to your problem above, is to save a copy of your model in the migration file (ie, a copy "frozen in time", so to speak), and use that copy in your migration, rather than your "real" one, which may be deleted down the track. Of course, you don't have to include a FULL copy of the ENTIRE model with all methods etc - you can remove anything that isn't needed for the specific migration.

This can make your migration files big and a bit messy, but who cares? Migrations aren't edited. They're run once and mostly forgotten. So it doesn't matter if they're a bit messy.

I haven't tried this second approach with sequelize, but it worked well for me with Rails.

Hope some of that's useful!

joshua.paling
  • 13,762
  • 4
  • 45
  • 60
  • Thanks for the thoughtful response. I messed around with these but found a new solution I like a little better. – Henry May 28 '17 at 22:26
  • More information for future users. I used the pattern #2: create a copy of the models at the migration level to have snapshot of what the database should look like when the migration is executing. – Jean-Francois Mar 25 '19 at 17:39
0

Thanks Josh for the pointers.

My final solution was to continue using Umzug and the same setup I've been using, and to replace any model based queries with raw queries using sequelize.query.

Sequelize provides some nice features for automatically formatting results like your model (JS object), so the only thing you have to do is write the queries. Mine are pretty simple inserts/updates/deletes.

This allows me to use the standard migration pattern for node, while still having reproducible migration history from app start to present, and not rely on changing models, and also use JS for my migrations instead of manipulating data in the SQL function language.

Hope this is helpful.

Henry
  • 491
  • 4
  • 13
0

(writing an answer as it has to much characters for a comment)

More information for future users.

I use a pattern similar to #2 from the answer of joshua.paling: create a copy of the models at the migration level to have a snapshot of what the database should look like when the migration is executing.

I ended up having some errors:

SequelizeAssociationError: You have used the alias children in two separate associations. Aliased associations must have unique aliases.

The solution is to clear the models and the associations between each execution of a migration.

function clearModels(sequelize) {
  Object.keys(sequelize.models).forEach(m => {
    Object.keys(sequelize.models[m].associations).forEach(a => {
      delete sequelize.models[m].associations[a];
    });
    delete sequelize.models[m];
  });
}
function clearCache(sequelize) {
  sequelize.importCache = {};
}

function migrated(sequelize) {
  return (name, migration) => {
    clearModels(sequelize);
    clearCache(sequelize);
  };
}

const {umzug, sequelize} = getUmzug();
umzug.on('migrated', migrated(sequelize));
umzug.on('reverted', migrated(sequelize));
Jean-Francois
  • 1,332
  • 3
  • 17
  • 24