Alter model to add "through" relationship to order a ManytoMany field - Django 1.7 migration modification

Question

I am trying to add an order to a ManyToMany field that I created a while ago. I basically want to order pictures in collections of pictures. I am running on Django 1.7, so no more South migrations (I was trying to follow this tutorial: http://mounirmesselmeni.github.io/2013/07/28/migrate-django-manytomany-field-to-manytomany-through-with-south/)

Here's the "through" relationship that I have:

class CollectionPictures(models.Model):
    picture = models.ForeignKey(
        Picture,
        verbose_name=u'Picture',
        help_text=u'Picture is included in this collection.',
    )
    collection = models.ForeignKey(
        Collection,
        verbose_name=u'Collection',
        help_text=u'Picture is included in this collection',
    )
    order = models.IntegerField(
        verbose_name=u'Order',
        help_text=u'What order to display this picture within the collection.',
        max_length=255
    )

    class Meta:
        verbose_name = u"Collection Picture"
        verbose_name_plural = u"Collection Pictures"
        ordering = ['order', ]

    def __unicode__(self):
        return self.picture.name + " is displayed in " + self.collection.name + (
        " in position %d" % self.order)


class Collection(models.Model):
    pictures = models.ManyToManyField(Picture, through='CollectionPictures', null=True)
    [... Bunch of irrelevant stuff after]

So this should work if I didn't have to migrate my old data (the only difference in the model is that it didn't have the through='CollectionPictures'

Here's my migration :

class Migration(migrations.Migration):

    dependencies = [
        ('artist', '0002_auto_20141013_1451'),
        ('business', '0001_initial'),
    ]

    operations = [
        migrations.CreateModel(
            name='CollectionPictures',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                ('order', models.IntegerField(help_text='What order to display this picture within the collection.', max_length=255, verbose_name='Order')),
                ('collection', models.ForeignKey(verbose_name='Collection', to='business.Collection', help_text='Picture is included in this collection')),
                ('picture', models.ForeignKey(verbose_name='Picture', to='artist.Picture', help_text='Picture is included in this collection.')),
            ],
            options={
                'ordering': ['order'],
                'verbose_name': 'Collection Picture',
                'verbose_name_plural': 'Collection Pictures',
            },
            bases=(models.Model,),
        ),
        migrations.AlterField(
            model_name='collection',
            name='pictures',
            field=models.ManyToManyField(to=b'artist.Picture', null=True, through='business.CollectionPictures'),
        ),
    ]

This throws an error when migrating:

ValueError: Cannot alter field business.Collection.pictures into business.Collection.pictures - they are not compatible types (you cannot alter to or from M2M fields, or add or remove through= on M2M fields)

Has anybody already tried that kind of manipulation with the new 1.7 migrations?

Thanks !

Just some links to similar questions: https://stackoverflow.com/q/33257530, https://stackoverflow.com/q/11466358, https://stackoverflow.com/q/6063357 — djvg, Jan 08 '19 at 16:13

score 23 · Accepted Answer · edited Jan 18 '15 at 20:27

23

The safest approach would be to create a new field and copy the data over.

Leave pictures alone and add pictures2 with your through field. Run makemigrations.
Edit the generated migration file and add a RunPython command where you copy data from the old table to the new table. Perhaps you can programmatically choose a good value for the new order column as well.
Delete the old pictures field. Run makemgirations.
Rename pictures2 to pictures. Run makemigrations.

This approach should leave you in the state you want with your data intact.

If copying over the data is a big problem you could try something else, like adding the order column in SQL, using the db_table option on CollectionPictures to make it point to the existing table, and then wiping out migrations and redoing with --fake. But that seems riskier than the approach above.

edited Jan 18 '15 at 20:27

blueyed

27,102
4
75
71

answered Oct 14 '14 at 03:55

Kevin Christopher Henry

46,175
7
116
102

I'm marking this as the right answer, as this is what I ended up doing before the topic got answered. If anybody steps on this, there's a work in progress here with django-sortedm2m https://github.com/gregmuellegger/django-sortedm2m/issues/7 – Alb Dum Oct 14 '14 at 05:06
Would there be any danger in doing it in a different order, viz. 2, 3, 1, so we could skip step 4? – djvg Jan 08 '19 at 14:50
@djvg: I'm not sure what you're suggesting. Step 2 above depends on step 1, so it couldn't come before. You're trying to avoid renaming? – Kevin Christopher Henry Jan 08 '19 at 22:30
Yes, indeed: Assuming the 'CollectionPictures' table has been migrated first, we could start by doing a data migration from the implicit through-table to this new table. Then we remove the 'pictures' field, without first creating a 'pictures2'. After migrating that, we can just add a new 'pictures' field with 'through=CollectionPictures'. Does that sound okay? – djvg Jan 08 '19 at 22:39

score 5 · Answer 2 · answered Feb 04 '18 at 15:17

Old question, but I had this problem too, and I've found a way with Django 1.11 that works, and should work with older versions too. The needed class exists back to 1.7 and still exists in 2.0

The fix involves manually changing the migration to do what we want, using the SeparateDatabaseAndState migration class. This class lets Django update the state, but gives us control over what operations to perform. In this case we just want to rename the model table, everything else is already set up right.

The steps:

Create your new ManyToMany Through model, but specify a custom table name, and no extra fields:

class CollectionPictures(models.Model):
    collection = ...
    picture = ...
    class Meta:
        # Change myapp to match.
        db_table = "myapp_collection_pictures"
        unique_together = (("collection", "picture"))

Taking the existing migration, and take the operations it generates and wrap it all in a single new SeparateDatabaseAndState:

class Migration(migrations.Migration):

    dependencies = [
        ('artist', '0002_auto_20141013_1451'),
        ('business', '0001_initial'),
    ]

    operations = [
        migrations.SeparateDatabaseAndState(
            database_operations=[
            ],
            state_operations=[
                migrations.CreateModel(
                    name='CollectionPictures',
                    fields=[
                        ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                        ('order', models.IntegerField(help_text='What order to display this picture within the collection.', max_length=255, verbose_name='Order')),
                        ('collection', models.ForeignKey(verbose_name='Collection', to='business.Collection', help_text='Picture is included in this collection')),
                        ('picture', models.ForeignKey(verbose_name='Picture', to='artist.Picture', help_text='Picture is included in this collection.')),
                    ],
                    options={
                        'ordering': ['order'],
                        'verbose_name': 'Collection Picture',
                        'verbose_name_plural': 'Collection Pictures',
                    },
                    bases=(models.Model,),
                ),
                migrations.AlterField(
                    model_name='collection',
                    name='pictures',
                    field=models.ManyToManyField(to=b'artist.Picture', null=True, through='business.CollectionPictures'),
                ),
            ]
        )

Remove the db_table from the class Meta, and add this operation after the SeparateDatabaseAndState, (not into the database_operations.):
```
migrations.AlterModelTable(
    name='collectionpicture',
    table=None,
),
```

Now if you run `./mange.py sqlmigrate myapp 0003 (pick the right number prefix!) you should with any luck see something like this as output:

BEGIN;
--
-- Custom state/database change combination
--
--
-- Rename table for collection[Pictures to None
--
ALTER TABLE "myapp_collection_pictures" RENAME TO "myapp_collectionpictures";
COMMIT;

Add your new columns ("order" in this case) and create a new migration. It's probably possible to do this at the same time, but I decided it was easier to do in two migrations.

(Step 3 isn't strictly required if you are happy keeping the custom table name there.)

And double check with ./manage.py makemigrations --check -- it should print "No changes detected".

Alter model to add "through" relationship to order a ManytoMany field - Django 1.7 migration modification

2 Answers2