I have an API built using Django and Django REST Framework that returns a paginated response to the front-end. By default the page_size is 25, but depending what instances of objects are being passed to the serializer it could either respond fairly quickly or very slowly.
The user can select a variety of parameters to make a search on. These parameters are then passed to the backend where the queryset is filtered. This queryset is then paginated:
paginator = Paginator(orders, length)
try:
orders_page = paginator.page(page_number)
except PageNotAnInteger:
orders_page = paginator.page(1)
except EmptyPage:
orders_page = paginator.page(paginator.num_pages)
Next the paginated queryset is serialized:
order_list = SerializerClass.setup_eager_loading(orders_page.object_list)
serializer = SerializerClass(order_list, many=True)
where
@staticmethod
def setup_eager_loading(queryset):
# select_related for "to-one" relationships
queryset = queryset.select_related('models', 'with', 'to-one', 'relationships')
queryset = queryset.prefetch_related('models', 'with', 'to-many', 'relationships')
and returned to the front-end:
response['data'] = serializer.data
response['draw'] = draw
response['recordsTotal'] = Order.objects.all().count()
response['recordsFiltered'] = paginator.count
return Response(response)
I understand that the filtering and pagination would take a variable amount of time depending on the parameters that are being filtered on, and the size of the queryset. I'm wondering why it is that the serializer has significantly different performance for different paginated querysets (anywhere from .5 seconds to 20 seconds).
My serializer class draws data from a lot of different parts of the database. I've used select_related, and prefetch_related in parts that were applicable, this improved the serializer performance across the board. There are a few SerializerMethodFields that still make queries resulting in N+1 issues, such as:
num_pieces = serializers.SerializerMethodField()
...
def get_num_pieces(self, o):
r = Pieces.objects.filter(set__order=o,
set__is_active=True,
is_active=True).count()
return r
this doesn't seem to be the major issue however. From close inspection the query that takes the longest to run is the query generated by the queryset = queryset.select_related('modelA', 'modelB', modelC','modelD','modelE'). I've also tried using the .only() function to select only the values needed in the query but this did not noticeably improve performance. My confusion is why the performance of this serializer could be so drastically different for querysets with the same number of objects.
