Django REST Framework: Setting up prefetching for nested serializers

Question

My Django-powered app with a DRF API is working fine, but I've started to run into performance issues as the database gets populated with actual data. I've done some profiling with Django Debug Toolbar and found that many of my endpoints issue tens to hundreds of queries in the course of returning their data.

I expected this, since I hadn't previously optimized anything with regard to database queries. Now that I'm setting up prefetching, however, I'm having trouble making use of properly prefetched serializer data when that serializer is nested in a different serializer. I've been using this awesome post as a guide for how to think about the different ways to prefetch.

Currently, my ReadingGroup serializer does prefetch properly when I hit the /api/readinggroups/ endpoint. My issue is the /api/userbookstats/ endpoint, which returns all UserBookStats objects. The related serializer, UserBookStatsSerializer, has a nested ReadingGroupSerializer.

The models, serializers, and viewsets are as follows:

models.py

class ReadingGroup(models.model):
    owner = models.ForeignKeyField(settings.AUTH_USER_MODEL)
    users = models.ManyToManyField(settings.AUTH_USER_MODEL)
    book_type = models.ForeignKeyField(BookType)
    ....
    <other group related fields>

   def __str__(self):
     return '%s group: %s' % (self.name, self.book_type)

class UserBookStats(models.Model):
    reading_group = models.ForeignKey(ReadingGroup)
    user = models.ForeignKey(settings.AUTH_USER_MODEL)
    alias = models.CharField()

    total_books_read = models.IntegerField(default=0)
    num_books_owned = models.IntegerField(default=0)
    fastest_read_time = models.IntegerField(default=0)
    average_read_time = models.IntegerField(default=0)

serializers.py

class ReadingGroupSerializer(serializers.ModelSerializer):
    users = UserSerializer(many = True,read_only=True)
    owner = UserSerializer(read_only=True)

    class Meta:
      model = ReadingGroup
      fields = ('url', 'id','owner', 'users')

    @staticmethod
    def setup_eager_loading(queryset):
      #select_related for 'to-one' relationships
      queryset = queryset.select_related('owner')

      #prefetch_related for 'to-many' relationships
      queryset = queryset.prefetch_related('users')

      return queryset

class UserBookStatsSerializer(serializers.HyperlinkedModelSerializer):
    reading_group = ReadingGroupSerializer()
    user = UserSerializer()
    awards = AwardSerializer(source='award_set', many=True)

    class Meta:
      model = UserBookStats
      fields = ('url', 'id', 'alias', 'total_books_read', 'num_books_owned', 
              'average_read_time', 'fastest_read_time', 'awards')

    @staticmethod
    def setup_eager_loading(queryset):
      #select_related for 'to-one' relationships
      queryset = queryset.select_related('user')

      #prefetch_related for 'to-many' relationships
      queryset = queryset.prefetch_related('awards_set')

      #setup prefetching for nested serializers
      groups = Prefetch('reading_group', queryset ReadingGroup.objects.prefetch_related('userbookstats_set'))        
      queryset = queryset.prefetch_related(groups)

      return queryset

views.py

class ReadingGroupViewset(views.ModelViewset):

  def get_queryset(self):
    qs = ReadingGroup.objects.all()
    qs = self.get_serializer_class().setup_eager_loading(qs)
    return qs

class UserBookStatsViewset(views.ModelViewset):

  def get_queryset(self):
    qs = UserBookStats.objects.all()
    qs = self.get_serializer_class().setup_eager_loading(qs)
    return qs

I've optimized the prefetching for the ReadingGroup endpoint (I actually posted about eliminating duplicate queries for that endpoint here), and now I'm working on the UserBookStats endpoint.

The issue I'm having is that, with my current setup_eager_loading in the UserBookStatsSerializer, it doesn't appear to use the prefetching set up by the eager loading method in the ReadingGroupSerializer. I'm still a little hazy on the syntax for the Prefetch object - I was inspired by this excellent answer to try that approach.

Obviously the get_queryset method of UserBookStatsViewset doesn't call setup_eager_loading for the ReadingGroup objects, but I'm sure there's a way to accomplish the same prefetching.

Are you asking about elegant solution or any solution? `queryset = queryset.prefetch_related('reading_group', 'reading_group__users', 'reading_group__owner')` should work just fine. — serg, Sep 24 '16 at 04:16
Any solution would be great - and wow, that's awesome. That works perfectly, so if you submit it as an answer, I'll accept. It replaces the `Prefetch` object, which is what I was hoping to use for an elegant solution that would sort of decouple prefetching, but this works really well. Thanks! — dkhaupt, Sep 24 '16 at 12:17

score 21 · Accepted Answer · answered Sep 24 '16 at 19:22

prefetch_related() supports prefetching inner relations by using double underscore syntax:

queryset = queryset.prefetch_related('reading_group', 'reading_group__users', 'reading_group__owner')

I don't think Django REST provides any elegant solutions out of the box for fetching all necessary fields automatically.

score 8 · Answer 2 · answered Dec 29 '19 at 16:33

An alternative to prefetching all nested relationships manually, there is also a package called django-auto-prefetching which will automatically traverse related fields on your model and serializer to find all the models which need to be mentioned in prefetch_related and select_related calls. All you need to do is add in the AutoPrefetchViewSetMixin to your ViewSets:

from django_auto_prefetching import AutoPrefetchViewSetMixin

class ReadingGroupViewset(AutoPrefetchViewSetMixin, views.ModelViewset):

  def get_queryset(self):
    qs = ReadingGroup.objects.all()
    return qs

class UserBookStatsViewset(AutoPrefetchViewSetMixin, views.ModelViewset):

  def get_queryset(self):
    qs = UserBookStats.objects.all()
    return qs

Any extra prefetches with more complex Prefetch objects can be added in the get_queryset method on the ViewSet.

Django REST Framework: Setting up prefetching for nested serializers

2 Answers2