Django Rest Framework- retrieving a related field on reverse foreign key efficiently

Question

I have the following models that represent a working group of users. Each working group has a leader and members:

class WorkingGroup(models.Model):
    group_name = models.CharField(max_length=255)
    leader = models.ForeignKey(User, null=True, on_delete=models.SET_NULL)

class WorkingGroupMember(models.Model):
    group = models.ForeignKey(WorkingGroup, on_delete=models.CASCADE)
    user = models.ForeignKey(User, on_delete=models.CASCADE)

In DRF, I want to efficiently retrieve all groups (there are several hundred) as an array of the following json objects:

{
    'id': <the_group_id>
    'group_name': <the_group_name>
    'leader': <id_of_leader>
    'members': [<id_of_member_1>, <id_of_member_2>, ...]
}

To do so, I have set up the following serializer:

class WorkingGroupSerializer(serializers.ModelSerializer):
    members = serializers.SerializerMethodField()
    class Meta:
        model = WorkingGroup
        fields = ('id', 'group_name', 'leader', 'members',)

    def get_members(self, obj):
        return obj.workinggroupmember_set.all().values_list('user_id', flat=True)

So that in my view, I can do something like:

groups = WorkingGroup.objects.all().prefetch_related('workinggroupmember_set')
group_serializer = WorkingGroupSerializer(groups, many=True)

This works, and gives the desired result, however I am finding it does not scale well at all, as the prefetching workinggroupmember_set does not seem to be used inside of the get_members method (Silky is showing a single query to grab all WorkingGroup objects, and then a query for each workinggroupmember_set call in the get_members method). Is there a way to set up the members field in the serializer to grab a flattened/single field version of workinggroupmember_set without using a SerializerMethodField? Or some other way of doing this that lets me properly use prefetch?

Are you open to changing you models? If yes, one solution could be to get rid of `WorkingGroupMember` completely and add a `ManyToManyField` to `User` called `members` directly under `WorkingGroup`. — slider, Oct 31 '18 at 22:59
Yes, that is definitely a possibility. In reality, the relationship is more like a many-to-many-through where the WorkingGroupMember has additional fields that are used in other areas of the app/api, but for this particular endpoint we simply want the users in that group. — MarkD, Nov 01 '18 at 00:41
A `ManyToManyField` will just create a transparent "through model" that works exactly like your `WorkingGroupMember` model. Your `ManyToManyField` may well use `WorkingGroupMember` as its through model and you would find yourself in the same situation. — A. J. Parr, Nov 01 '18 at 04:03

score 4 · Accepted Answer · answered Nov 01 '18 at 04:09

Problem here that you are doing values_list on top of all which nullifies your prefetch_related. There is currently no way to do prefetch with values_list see https://code.djangoproject.com/ticket/26565. What you can do is to transition this into python code instead of SQL

class WorkingGroupSerializer(serializers.ModelSerializer):
    members = serializers.SerializerMethodField()
    class Meta:
        model = WorkingGroup
        fields = ('id', 'group_name', 'leader', 'members',)

    def get_members(self, obj):
        return [wgm.user_id for wgm in obj.workinggroupmember_set.all()]

Brilliant! Thanks for pointing this issue out- I've been looking all over the DRF docs for why this wasn't working, and it turns out to be a django issue instead. Your fix worked perfect. — MarkD, Nov 01 '18 at 12:38

score 0 · Answer 2 · answered Feb 17 '19 at 23:44

In a recent project with DRF v3.9.1 and django 2.1, I needed to recursively expose all the children of an object, by having only a direct connection to the parent, which could have had multiple children.

Before, if I was to request the "tree" of an object, I was getting:

{
    "uuid": "b85385c0e0a84785b6ca87ce50132659",
    "name": "a",
    "parent": null
}

By applying the serialization shown below I get:

{
    "uuid": "b85385c0e0a84785b6ca87ce50132659",
    "name": "a",
    "parent": null
    "children": [
        {
            "uuid": "efd26a820b4e4f7c8e56c812a7791fcb",
            "name": "aa",
            "parent": "b85385c0e0a84785b6ca87ce50132659"
            "children": [
                {
                    "uuid": "ca2441fc7abf49b6aa1f3ebbc2dae251",
                    "name": "aaa",
                    "parent": "efd26a820b4e4f7c8e56c812a7791fcb"
                    "children": [],
                }
            ],
        },
        {
            "uuid": "40e09c85775d4f1a8578bba9c812df0e",
            "name": "ab",
            "parent": "b85385c0e0a84785b6ca87ce50132659"
            "children": [],
        }
    ],
}

Here is the models.py of the recursive object:

class CategoryDefinition(BaseModelClass):
    name = models.CharField(max_length=100)
    parent = models.ForeignKey('self', related_name='children',
                               on_delete=models.CASCADE,
                               null=True, blank=True)

To get all the reverse objects in the foreign key, apply a field to the serializer class:

class DeepCategorySerializer(serializers.ModelSerializer):
    children = serializers.SerializerMethodField()

    class Meta:
        model = models.CategoryDefinition
        fields = '__all__'

    def get_children(self, obj):
        return [DeepCategorySerializer().to_representation(cat) for cat in obj.children.all()]

Then apply this serializer to a DRF view function or generics class, such as:

re_path(r'categories/(?P<pk>[\w\d]{32})/',
        generics.RetrieveUpdateDestroyAPIView.as_view(
            queryset=models.CategoryDefinition.objects.all(),
            serializer_class=serializers.DeepCategorySerializer),
        name='category-update'),

Django Rest Framework- retrieving a related field on reverse foreign key efficiently

2 Answers2

Linked