16

I'm having trouble refactoring a superclass in Django v2.2.12 involving three models, with one superclass model and two subclass models:

class BaseProduct(models.Model):
    name = models.CharField()
    description = models.CharField()


class GeneralProduct(BaseProduct):
    pass


class SoftwareProduct(BaseProduct):
    pass

The BaseProduct model needs to be renamed to just Product, so I changed this code to:

class Product(models.Model):
    name = models.CharField()
    description = models.CharField()

class GeneralProduct(Product):
    pass


class SoftwareProduct(Product):
    pass

And then ran python manage.py makemigrations, in which Django seems to correctly see what changed:

Did you rename the yourapp.BaseProduct model to Product? [y/N] y
Did you rename generalproduct.baseproduct_ptr to generalproduct.product_ptr (a OneToOneField)? [y/N] y
Did you rename softwareproduct.baseproduct_ptr to softwareproduct.product_ptr (a OneToOneField)? [y/N] y

Migrations for 'yourapp':
  .../yourapp/migrations/002_auto_20200507_1830.py
    - Rename model BaseProduct to Product
    - Rename field baseproduct_ptr on generalproduct to product_ptr
    - Rename field baseproduct_ptr on softwareproduct to product_ptr

So far so good. Django sees that the superclass got renamed, and it knows that its own autogenerated ..._ptr values that it uses to track model inheritance need to be updated in the db, too.

The resulting migration it comes up with looks about as terse as it should be:

# Generated by Django 2.2.12 on 2020-05-07 18:30

from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ('yourapp', '0001_initial'),
    ]

    operations = [
        migrations.RenameModel(
            old_name='BaseProduct',
            new_name='Product',
        ),
        migrations.RenameField(
            model_name='generalproduct',
            old_name='baseproduct_ptr',
            new_name='product_ptr',
        ),
        migrations.RenameField(
            model_name='softwareproduct',
            old_name='baseproduct_ptr',
            new_name='product_ptr',
        ),
    ]

This all looks perfect, but applying that migration using python manage.py migrate crashes out:

Running migrations:
  Applying yourapp.0002_auto_20200507_1830...Traceback (most recent call last):
  [...]
  File ".../python3.7/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File ".../python3.7/site-packages/django/db/migrations/migration.py", line 114, in apply
    operation.state_forwards(self.app_label, project_state)
  File ".../python3.7/site-packages/django/db/migrations/operations/models.py", line 340, in state_forwards
    state.reload_models(to_reload, delay=True)
  File ".../python3.7/site-packages/django/db/migrations/state.py", line 165, in reload_models
    self._reload(related_models)
  File ".../python3.7/site-packages/django/db/migrations/state.py", line 191, in _reload
    self.apps.render_multiple(states_to_be_rendered)
  File ".../python3.7/site-packages/django/db/migrations/state.py", line 308, in render_multiple
    model.render(self)
  File ".../python3.7/site-packages/django/db/migrations/state.py", line 579, in render
    return type(self.name, bases, body)
  File ".../python3.7/site-packages/django/db/models/base.py", line 253, in __new__
    base.__name__,
django.core.exceptions.FieldError: Auto-generated field 'baseproduct_ptr' in class 'SoftwareProduct' for
parent_link to base class 'BaseProduct' clashes with declared field of the same name.

I searched the web for that error, as well as for renaming a Django model that's a superclass for other models, but there does not appear to be any (discoverable) documentation, or blog posts, or SO answers, that talk about this problem.

Mike 'Pomax' Kamermans
  • 49,297
  • 16
  • 112
  • 153
  • 1
    Would you mind adding the relevant fields too? (maybe that would help us to reproduce the behavior ) – JPG May 10 '20 at 04:38
  • 1
    **FYI:** I have created a similar scenario, but it didn't throw any errors. So, it's nice to add ***"minimal reproducable"*** code sample – JPG May 10 '20 at 04:58
  • @ArakkalAbu This **is** a MRE... Add an arbitrary field to the `BaseProduct` model to avoid errors. – Lord Elrond May 10 '20 at 15:58
  • I don't think your example is a MRE one, (I was succeeded running migrations in this config, [Before](https://i.stack.imgur.com/3V2NQ.png) and [After](https://i.stack.imgur.com/Oq6Ja.png) – JPG May 10 '20 at 17:20
  • 3
    Note that `BaseProduct` is not abstract, though. Any fields should be fine, but `Meta` can change what Django does, so try to steer clear of that. Also, do not add that pointer: you're still showing code with fields that Django manages. You never add those, they simply exist: by making them `Charfield` you've completely negated what this question is asking about. I've added some fields so you can stop being confused about what is being asked: Django _makes_ `superclassname_ptr` fields to track model inheritance, and the problem is about what happens when _it_ writes code to change those. – Mike 'Pomax' Kamermans May 10 '20 at 18:27

1 Answers1

15

Canonical answer

The reason this goes wrong is that even though Django sees that a model got renamed and subclasses need pointer updates, it can't properly execute on those updates. There is a PR open to add this to Django as of the time of writing (https://github.com/django/django/pull/13021, originally 11222), but until that lands, the solution is to temporarily "trick" Django into thinking the subclasses are in fact plain models without any inheritance, and effecting the changes manually by running through the following steps:

  1. renaming the autogenerated inheritance pointer from superclass_ptr to newsuperclass_ptr manually (in this case, baseproduct_ptr becomes product_prt), then
  2. trick Django into thinking the subclasses are just generic Model implementations by literally rewriting the .bases property for them and telling Django to reload them, then
  3. renaming the superclass to its new name (in this case, BaseProduct becomes Product) and then finally
  4. updating the newsuperclass_ptr fields so that they point to the new superclass name instead, making sure to specify auto_created=True as well as parent_link=True.

In the last step, the first property should be there mostly because Django auto-generates pointers, and we don't want Django to be able to tell we ever tricked it and did our own thing, and the second property is there because parent_link is the field that Django relies on to correctly hook up model inheritance when it runs.

So, a few more steps than just manage makemigrations, but each is straight-forward, and we can do all of this by writing a single, custom migration file.

Using the names from the question post:

# Custom Django 2.2.12 migration for handling superclass model renaming.

from django.db import migrations, models
import django.db.models.deletion

# with a file called custom_operations.py in our migrations dir:
from .custom_operations import AlterModelBases


class Migration(migrations.Migration):
    dependencies = [
        ('yourapp', '0001_initial'),
        # Note that if the last real migration starts with 0001,
        # this migration file has to start with 0002, etc. 
        #
        # Django simply looks at the initial sequence number in
        # order to build its migration tree, so as long as we
        # name the file correctly, things just work.
    ]

    operations = [
        # Step 1: First, we rename the parent links in our
        # subclasses to match their future name:

        migrations.RenameField(
            model_name='generalproduct',
            old_name='baseproduct_ptr',
            new_name='product_ptr',
        ),
        
        migrations.RenameField(
            model_name='softwareproduct',
            old_name='baseproduct_ptr',
            new_name='product_ptr',
        ),

        # Step 2: then, temporarily set the base model for
        #         our subclassses to just `Model`, which makes
        #         Django think there are no parent links, which
        #         means it won't try to apply crashing logic in step 3.

        AlterModelBases("GeneralProduct", (models.Model,)),
        AlterModelBases("SoftwareProduct", (models.Model,)),

        # Step 3: Now we can safely rename the superclass without
        #         Django trying to fix subclass pointers:

        migrations.RenameModel(
            old_name="BaseProduct",
            new_name="Product"
        ),

        # Step 4: Which means we can now update the `parent_link`
        #         fields for the subclasses: even though we altered
        #         the model bases earlier, this step will restore
        #         the class hierarchy we actually need:

        migrations.AlterField(
            model_name='generalproduct',
            name='product_ptr',
            field=models.OneToOneField(
                auto_created=True,
                on_delete=django.db.models.deletion.CASCADE,
                parent_link=True, primary_key=True,
                serialize=False,
                to='buyersguide.Product'
            ),
        ),

        migrations.AlterField(
            model_name='softwareproduct',
            name='product_ptr',
            field=models.OneToOneField(
                auto_created=True,
                on_delete=django.db.models.deletion.CASCADE,
                parent_link=True,
                primary_key=True,
                serialize=False,
                to='buyersguide.Product'
            ),
        ),
    ]

The crucial step is the inheritance "destruction": we tell Django that the subclasses inherit from models.Model, so that renaming the superclass will leave the subclasses entirely unaffected (rather than Django trying to update inheritance pointers itself), but we do not actually change anything in the database. We only make that change to the currently running code, so if we exit Django, it'll be as if that change was never made to begin with.

So to achieve this, we're using a custom ModelOperation that can change the inheritance of a(ny) class to a(ny collection of) different superclass(es) at runtime:

# contents of yourapp/migrations/custom_operations.py

from django.db.migrations.operations.models import ModelOperation


class AlterModelBases(ModelOperation):
    reduce_to_sql = False
    reversible = True

    def __init__(self, name, bases):
        self.bases = bases
        super().__init__(name)

    def state_forwards(self, app_label, state):
        """
        Overwrite a models base classes with a custom list of
        bases instead, then force Django to reload the model
        with this (probably completely) different class hierarchy.
        """
        state.models[app_label, self.name_lower].bases = self.bases
        state.reload_model(app_label, self.name_lower)

    def database_forwards(self, app_label, schema_editor, from_state, to_state):
        pass

    def database_backwards(self, app_label, schema_editor, from_state, to_state):
        pass

    def describe(self):
        return "Update %s bases to %s" % (self.name, self.bases)

With this custom migration file and our custom_operations.py in place, all we need to do is update our code to reflect the new naming scheme:

class Product(models.Model):
    name = models.CharField()
    description = models.CharField()

class GeneralProduct(Product):
    pass


class SoftwareProduct(Product):
    pass

And then apply manage migrate, which will run and update everything as needed.

NOTE: depending on whether or not you "prefactored" your code in preparation for your renaming, using something like this:

class BaseProduct(models.Model):
    name = models.CharField()
    description = models.CharField()


# "handy" aliasing so that all code can start using `Product`
# even though we haven't renamed actually renamed this class yet:
Product = BaseProduct


class GeneralProduct(Product):
    pass


class SoftwareProduct(Product):
    pass

you may have to update ForeignKey and ManyToMany relations to Productin other classes, add explicit add models.AlterField instructions to update BaseProduct to Product:

        ...
        migrations.AlterField(
            model_name='productrating',
            name='product',
            field=models.ForeignKey(
                 on_delete=django.db.models.deletion.CASCADE,
                 to='yourapp.Product'
            ),
        ),
        ...

 


 

Original answer

Oh, yea, this is a tricky one. But I've solved in my project here is the way I did it.

1) Delete newly created migration and rollback your model changes

2) Change your implicit parent link fields to explicit with parent_link option. We need this to manually rename our field to propper name in later steps

class BaseProduct(models.Model):
    ...

class GeneralProduct(BaseProduct):
    baseproduct_ptr = models.OneToOneField(BaseProduct, django.db.models.deletion.CASCADE, parent_link=True, primary_key=True)

class SoftwareProduct(BaseProduct):
    baseproduct_ptr = models.OneToOneField(BaseProduct, django.db.models.deletion.CASCADE, parent_link=True, primary_key=True)

3) Generate migration via makemigrations and get something like this

...
migrations.AlterField(
    model_name='generalproduct',
    name='baseproduct_ptr',
    field=models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, parent_link=True, primary_key=True, serialize=False, to='BaseProduct'),
),
migrations.AlterField(
    model_name='softwareproduct',
    name='baseproduct_ptr',
    field=models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, parent_link=True, primary_key=True, serialize=False, to='BaseProduct'),
)
...

4) Now you have explicit links to your parent model you could rename them to product_ptr which will match your desired link name

class GeneralProduct(BaseProduct):
    product_ptr = models.OneToOneField(BaseProduct, django.db.models.deletion.CASCADE, parent_link=True, primary_key=True)

class SoftwareProduct(BaseProduct):
    product_ptr = models.OneToOneField(BaseProduct, django.db.models.deletion.CASCADE, parent_link=True, primary_key=True)

5) Generate migration via makemigrations and get something like this

...
migrations.RenameField(
    model_name='generalproduct',
    old_name='baseproduct_ptr',
    new_name='product_ptr',
),
migrations.RenameField(
    model_name='softwareproduct',
    old_name='baseproduct_ptr',
    new_name='product_ptr',
),
...

6) Now the trickiest part we need to add new migration operation(source could be found here https://github.com/django/django/pull/11222) and put in our code, I personally have contrib package in my project where I put all staff like this

File in contrib/django/migrations.py

# https://github.com/django/django/pull/11222/files
# https://code.djangoproject.com/ticket/26488
# https://code.djangoproject.com/ticket/23521
# https://code.djangoproject.com/ticket/26488#comment:18
# https://github.com/django/django/pull/11222#pullrequestreview-233821387
from django.db.migrations.operations.models import ModelOperation


class DisconnectModelBases(ModelOperation):
    reduce_to_sql = False
    reversible = True

    def __init__(self, name, bases):
        self.bases = bases
        super().__init__(name)

    def state_forwards(self, app_label, state):
        state.models[app_label, self.name_lower].bases = self.bases
        state.reload_model(app_label, self.name_lower)

    def database_forwards(self, app_label, schema_editor, from_state, to_state):
        pass

    def database_backwards(self, app_label, schema_editor, from_state, to_state):
        pass

    def describe(self):
        return "Update %s bases to %s" % (self.name, self.bases)

7) Now we are ready to rename our parent model

class Product(models.Model):
    ....

class GeneralProduct(Product):
    pass


class SoftwareProduct(Product):
    pass

8) Generate migration via makemigrations. Make sure you add DisconnectModelBases step, it will not be added automatically, even if successfully generate migration. If this doesn't help and you could try creating --empty one manully.

from django.db import migrations, models
import django.db.models.deletion

from contrib.django.migrations import DisconnectModelBases


class Migration(migrations.Migration):

    dependencies = [
        ("contenttypes", "0002_remove_content_type_name"),
        ("products", "0071_auto_20200122_0614"),
    ]

    operations = [
        DisconnectModelBases("GeneralProduct", (models.Model,)),
        DisconnectModelBases("SoftwareProduct", (models.Model,)),
        migrations.RenameModel(
            old_name="BaseProduct", new_name="Product"
        ),
        migrations.AlterField(
            model_name='generalproduct',
            name='product_ptr',
            field=models.OneToOneField(auto_created=True, on_delete=django.db.models.deletion.CASCADE, parent_link=True, primary_key=True, serialize=False, to='products.Product'),
        ),
        migrations.AlterField(
            model_name='softwareproduct',
            name='product_ptr',
            field=models.OneToOneField(auto_created=True, on_delete=django.db.models.deletion.CASCADE, parent_link=True, primary_key=True, serialize=False, to='proudcts.Product'),
        ),
    ]

NOTE: after all this, you don't need explicit parent_link fields. So you can remove them. Which I actually did on step 7.

Mike 'Pomax' Kamermans
  • 49,297
  • 16
  • 112
  • 153
Sardorbek Imomaliev
  • 14,861
  • 2
  • 51
  • 63
  • Trying this right now, so a few questions: can you explain why `AlterModelBases` has to come before the superclass rename? (specifically to make sure that as canonical answer, that information is captured). Also, could you explain (probably in code comments?) why `AlterModelBases.state_forward` works? – Mike 'Pomax' Kamermans May 11 '20 at 18:51
  • Also, I've taken your approach and reduced it to a single migration file with four steps, obviating the need to make the parent class ptr field explicit, so I'd like to edit this answer a bit before marking it as the correct answer, just to make sure we have a canonical answer that consists of as few steps as necessary. – Mike 'Pomax' Kamermans May 11 '20 at 19:53
  • 1
    You may note also, that overriding `state_forwards` method of `migrations.RenameModel` object like this [example](https://code.djangoproject.com/ticket/26488#comment:17) works as expected within django 2.2.12 for most databases except SQLite <= 3.26. – Chiheb Nexus May 11 '20 at 21:13
  • @Mike'Pomax'Kamermans the way `AlterModelBases` is implemented in patch doesn't fit its name, it probably should be name `DisconnectModelBase` (https://github.com/django/django/pull/11222#pullrequestreview-233821387) also if you look into the source code you fill see that it calls`state.reload_model` which actually recollects model dependencies. Answering your second question why it has to come before `RenameModel`, initially I just copied solution from here https://code.djangoproject.com/ticket/26488#comment:18. But the first part of this comment explains why it should come first – Sardorbek Imomaliev May 12 '20 at 06:07
  • I suggest not deleting my original answer, but adding a more concise variant on at the top of it. This way people will have a better understanding of what the underlying issue is and the approach required to solve it. Also, you could create a separate answer containing your simplified solution. – Sardorbek Imomaliev May 12 '20 at 06:10
  • @Mike'Pomax'Kamermans updated my answer slightly renamed `AlterModelBases` to `DisconnectModelBases` which will make it more understandable. – Sardorbek Imomaliev May 12 '20 at 06:14
  • The rewrite would preserve the explanation of the underlying issue (that's part of the bounty, after all) - I'd rephrase it as a problem statement first, then show the single migration with plenty of code comments that explain each step. It is functionally the same as your individual steps, but does not require constantly updating the .py file(s) and rerunning makemigrations. So your original concepts will still be there, just made clearer by having fewer actions the user has to take, so that it's a better canonical answer. And yes, that name make it clearer what's happening, thanks. – Mike 'Pomax' Kamermans May 12 '20 at 14:58
  • I've taken your answer, and left it untouched as the "original answer", but turned it into a canonical answer above that, which turns the process into a single migration file consisting of four steps, so that you're not updating code just to run `makemigrations` several time: just create the migration file with the correct sequence number in the filename and previous one in the dependency list, then it's basically "follow the recipe", save, and run. Thank you very much for the help, as well as volunteering to see if you can help get that PR over the finishing line. You 110% deserve this bounty – Mike 'Pomax' Kamermans May 12 '20 at 22:49
  • 1
    Thanks, glad to help)) This issue would have taken any developer too much time without conical solution. I myself if I recall spent 2 or 3 days to find a solution when I needed this in my project. – Sardorbek Imomaliev May 13 '20 at 00:04
  • 2
    no kidding, I got stuck on this for 2 days before going "this is silly, I have too much rep already, and someone _must_ have solved this already". Thank you for finding this and being willing to help get everyone a good answer to refer to! – Mike 'Pomax' Kamermans May 14 '20 at 00:55