Django many-to-many add having inconsistent effect in functions, works on console

Question

I'm creating a lesson planning tool. Each lesson covers multiple syllabus points, and has associated resources. When you save a lesson, I want to make sure that the syllabus points for the lesson are also saved to the resources.

I've tried using both signals and overriding the save method to make this work, but keep getting weird results.

models.py

class Lesson(models.Model):
    lessonslot = models.ForeignKey(TimetabledLesson, on_delete=models.CASCADE)
    classgroup = models.ForeignKey(ClassGroup, null=True, blank=False, on_delete=models.SET_NULL)
    status = models.CharField(max_length=20, null=True, blank=True)
    syllabus_points_covered = models.ManyToManyField(SyllabusPoint, blank=True)
    lesson_title = models.CharField(max_length=200, null=True, blank=True)
    description = models.TextField(null=True, blank=True)
    requirements = models.TextField(null=True, blank=True)
    sequence = models.IntegerField(null=False, blank=True)
    date = models.DateField(null=True, blank=True)
    syllabus = models.ForeignKey(Syllabus, blank=True, null=True, on_delete=models.SET_NULL)

    class Meta:
        unique_together = ( ("lessonslot", "date"),
                           ("classgroup", "sequence"))

class LessonResources(models.Model):
    lesson = models.ForeignKey(Lesson, blank=True, null=True, on_delete=models.SET_NULL)
    resource_type = models.CharField(max_length=100, choices=RESOURCE_TYPES, null=False, blank=False)
    resource_name = models.CharField(max_length=100, null=True, blank=False)
    link = models.URLField(blank=True, null=True)
    students_can_view_before = models.BooleanField()
    students_can_view_after = models.BooleanField()
    available_to_all_classgroups = models.BooleanField()
    syllabus_points = models.ManyToManyField(SyllabusPoint, blank=True)

    def set_syllabus_points(self):
        if self.lesson:
            points = self.lesson.syllabus_points_covered.all().order_by('pk')
            for point in points:
                self.syllabus_points.add(point)
            print ('In set_syllabus_points')
            print(self.syllabus_points.all())
        return self

signals.py

from timetable.models import LessonResources, Lesson
from django.db.models.signals import post_save, m2m_changed
from django.dispatch import receiver


@receiver(m2m_changed, sender=Lesson.syllabus_points_covered.through)
def post_update_lesson_syllabus_pts(sender, instance, **kwargs):
    """ After adding a syllabus point to a lesson, update its resources"""
    resources = instance.resources()
    for resource in resources:
        resource.set_syllabus_points()


@receiver(post_save, sender=LessonResources)
def update_resources(sender, instance, **kwargs):
""" After adding a resource, check its syllabus points match its lesson """
    print('Before set_syllabus_points')
    print(instance.pk)
    print(instance.syllabus_points.all())
    instance.set_syllabus_points()
    print('After set_syllabus_points')
    print(instance.syllabus_points.all())

Console Output

Before set_syllabus_points
105
<QuerySet []>
After set_syllabus_points
<QuerySet [<SyllabusPoint: Motion and Measurement 1.1.1 Use and describe the use of rules and measuring cylinders to find a length or a volume>, <SyllabusPoint: Motion and Measurement 1.1.3 Obtain an average value for a small distance and for a short interval of time by measuring multiples (including the period of a pendulum)>, <SyllabusPoint: Motion and Measurement 1.1.3 Understand that a micrometer screw gauge is used to measure very small distances>, <SyllabusPoint: Motion and Measurement 1.2.1 Define speed>, <SyllabusPoint: Motion and Measurement 1.2.2 Calculate Average speed from total distance / total time>, <SyllabusPoint: Forces 1.3.1 Explain that In the absence of an unbalanced force, an object will either remain at rest or travel with a constant speed in a straight line. Unbalanced forces change motion.>]>

The first one (post_update_lesson_syllabus_pts) works fine, but adding the points to a resource after that resource has been created does not work.

In the debugger, I can see that resource.set_syllabus_points() is being called after a new resource is created or modified. I can also see that self.syllabus_points.add(point) is passing a valid syllabuspoint as point. However, when I then check the resource (e.g. print(resources.syllabus_points.all()) after eveyrthing has completed, I get an empty queryset! I'm creating the resource in the admin interface (using an inline formset attached to the lesson), and none of the syllabus points are selected.

When the function runs, the print statement in set_syllabus_points outputs the correct queryset - why aren't they getting to the database?

Thanks in advance for all your help, I'm sure it's something silly I've missed.

UPDATE 1

Okay, so I think I've narrowed it down to something that's happening in the admin interface.

I've added some print statements shown above to see what's going on.

Here's the output from running an interractive shell:

>>>> resource = LessonResources.objects.get(pk=115)
>>>> resource.syllabus_points.all()
<QuerySet []>
>>>> resource.set_syllabus_points()
In set_syllabus_points
<QuerySet [<SyllabusPoint: Motion and Measurement 1.1.1 Use and describe the use of rules and measuring cylinders to find a length or a volume>, <SyllabusPoint: Motion and Measurement 1.1.3 Obtain an average value for a small distance and for a short interval of time by measuring multiples (including the period of a pendulum)>, <SyllabusPoint: Motion and Measurement 1.1.3 Understand that a micrometer screw gauge is used to measure very small distances>, <SyllabusPoint: Motion and Measurement 1.2.1 Define speed>, <SyllabusPoint: Motion and Measurement 1.2.2 Calculate Average speed from total distance / total time>, <SyllabusPoint: Forces 1.3.1 Explain that In the absence of an unbalanced force, an object will either remain at rest or travel with a constant speed in a straight line. Unbalanced forces change motion.>]>
>>>> resource.syllabus_points.all()
<QuerySet [<SyllabusPoint: Motion and Measurement 1.1.1 Use and describe the use of rules and measuring cylinders to find a length or a volume>, <SyllabusPoint: Motion and Measurement 1.1.3 Obtain an average value for a small distance and for a short interval of time by measuring multiples (including the period of a pendulum)>, <SyllabusPoint: Motion and Measurement 1.1.3 Understand that a micrometer screw gauge is used to measure very small distances>, <SyllabusPoint: Motion and Measurement 1.2.1 Define speed>, <SyllabusPoint: Motion and Measurement 1.2.2 Calculate Average speed from total distance / total time>, <SyllabusPoint: Forces 1.3.1 Explain that In the absence of an unbalanced force, an object will either remain at rest or travel with a constant speed in a straight line. Unbalanced forces change motion.>]>

We also seem to be seeing the correct output when set_syllabus_points() is called by signals due to creating or updating a resource.

However, even though our console log is showing it's added the points, they seems to re-set back to an empty queryset again at some points as the admin interface is loaded. Could this be caused by something in the admin views / forms caching the many-to-many relations and re-setting then back to their initial values?

UPDATE 2

As suspected, it's definitely a modelAdmin thing. This StackOverflow post and this blog post explain what's happening - now I just have to work out how to override the default behaviour for a tabularInLine form...

Answer 1

In your post_update_lesson_syllabus_pts add statement resource.save() once your have called set_syllabus_points(). Post that you should be able to see the syllabus points for your resource.

Answer 2

If what you mean by "I got nothing returned" is that you got an object that looked something like this:

<django.db.models.fields.related_descriptors.create_forward_many_to_many_manager.<locals>.ManyRelatedManager object at 0x1111eb518>

Then that's not what you think it is. When you execute resource.syllabus_points, what you're doing is accessing that LessonResources instance's related SyllabusPoints objects through a ManyToMany manager. You'd be returned a similar object, with less constraints, if you called SyllabusPoints.objects. Much like you would with any other model manager, you need to call something like all, get, or filter on that manager. In this example, all I have at hand is a Blog app where I have a blog.Post model with a ManyToManyField to blog.Tag. It's all the same though:

>>> from naguine.apps import get_model
>>> Post = get_model('blog.Post')
>>> Tag = get_model('blog.Tag')
>>> p = Post.objects.create()
>>> p.tags.add(Tag.objects.create(name='testtag'))
>>> p.tags
<django.db.models.fields.related_descriptors.create_forward_many_to_many_manager.<locals>.ManyRelatedManager object at 0x1111eb518>
>>> p.tags.all()
<QuerySet [<Tag: testtag>]>

Answer 3

As noted in Update 2, this issue was caused by the way that the Django admin interface handles many-to-many relations.

The django admin interface was using a TabularInline admin form so that I could edit lessons and their associated resources at one;

admin.py

from django.contrib import admin
from timetable.models import *
from timetable.forms import LessonForm
# Register your models here.


class ResourcesInLine(admin.TabularInline):
    model = LessonResources

class LessonAdmin(admin.ModelAdmin):
    form = LessonForm
    inlines = [
        ResourcesInLine,
    ]


admin.site.register(LessonSlot)
admin.site.register(TimetabledLesson)
admin.site.register(Lesson, LessonAdmin)
admin.site.register(LessonResources)
admin.site.register(LessonSuspension)

When I saved the admin form, the basic order of what was happening was:

1. The lesson details were saved.
2. Resources were saved, ignoring the m2m relations.
3. Saving the resource triggered the post-save signal for set_syllabus_points()
4. Syllabus points were added to the resource.
5. The admin module cleared all the m2m relations, as noted in the Django documentation.
6. The admin module added each of the syllabus points as shown in the admin interface.

It was step 5 that was the problem. When I saved the lesson form, the syllabus_points for the lesson resources were all blank. The signal worked as expected and added them, but the admin module then cleared them again because its expected behaviour was to make the LessonResoure instance fit what was on the initial form - which was nothing!

My long-term solution to this will be to build a custom form outside the admin interface to do this myself. Since this won't call the clean method on the many-to-many relation, this problem will stop.

In the meantime, I've removed syllabus_points from the admin interface for creating a new lesson. This means that the clear method doesn't get called, and fixed the issue.

Fix Edit the admin interface to remove the many-to-many relation, e.g;

from django.contrib import admin
from timetable.models import *
from timetable.forms import LessonForm
# Register your models here.


class ResourcesInLine(admin.TabularInline):
    model = LessonResources
    exclude = ('syllabus_points',)

class LessonAdmin(admin.ModelAdmin):
    form = LessonForm
    inlines = [
        ResourcesInLine,
    ]


admin.site.register(LessonSlot)
admin.site.register(TimetabledLesson)
admin.site.register(Lesson, LessonAdmin)
admin.site.register(LessonResources)
admin.site.register(LessonSuspension)

Huge thanks to @Cole and @Rajesh for pointing me in the right directions!