Django rest framework ModelSerializer runs too slowly

Question

We want to provide an api for our django project,so we use drf (django rest framework). we use ModelSerializer which provides a shortcut that lets you automatically create a Serializer class with fields that correspond to the Model fields. Our problem is it runs very slowly.In other word,process of serialization takes about 40 second until retrieving the response.

How we can reduce this delay?
VIEW

class MyObjViewSet(viewsets.ModelViewSet):

    pagination_class = LargeResultsSetPagination

    def get_queryset(self):
        queryset = MyObj.objects.all().order_by('time')
        return queryset

    serializer_class = MyObjSerializer

My Obj Model

class MyObj(models.Model):
    id = models.BigIntegerField(primary_key=True)
    time = models.DateTimeField()
    type = models.CharField(max_length=5)
    user = models.ForeignKey('User', related_name='myobj')

MyObj User Model

class User(models.Model):
    id = models.IntegerField(primary_key=True)
    username = models.CharField(max_length=80)

my Obj serializer

class MyObjSerializer(serializers.ModelSerializer):

    class Meta:
        model = MyObj
        fields = ('id', 'type', 'time', 'user')

My problem is when i want to retrieve list of my objs, its take about to 40 second!

Answer 1

Actualy, it is a real problem of drf.
If you have a lot of objects returned by queryset then you have to do few steps:
1. Check that you use many=True, it will produce significant performance improvement.
2. Avoid ModelSerializer on huge objects sets. In fact, worth idea is to avoid any kind of rest_framework.Serializer
3. Try to use serpy library for serializing. But you shouldn't forget that it is not fully compatible with rest_framework.Serializer.
4. Use .values() and DictSerializer. It will give you BIG speed up of serializing.
5. Do not forget about indexes in your database. 6. Use such powerful things as prefetch_related and select_related while working with ForeignKey.
7. The last way is to use simple dict. Otherwise, I didn't get appreciable result: only 10% versus serpy with DictSerializer.

I had a case when I have to serialize many objects (about 3-5k), overhead from drf serializer was at least 2,5 seconds (without sql's time). After optimizing I got about ~200-300 ms.

I hope, drf's developers will do some perfomance-improvements in framework.

Answer 2

Man, don't use "MyObj.objects.all().order_by('time')",

If you has above tens of hundreds of datas, then just get all and order by time, it would takes so long......

"that is not the seriallizer's problem, that is an order problem." you can limit the time you search, use:

1、gt：bigger then

now = datetime.datetime.now()
#yesterday
start = now – datetime.timedelta(hours=23, minutes=59, seconds=59)
a=yourobject.objects .filter(youdatetimcolumn__gt=start)

2、gte：bigger or equl then

a=yourobject.objects .filter(youdatetimcolumn__gte=start)

3、lt：littler then

a=yourobject.objects .filter(youdatetimcolumn__lt=start)

4、lte：littler or equal then

a=yourobject.objects .filter(youdatetimcolumn__lte=start)

5、range： a range of time

start_date = datetime.date(2005, 1, 1)
end_date = datetime.date(2005, 3, 31)
Entry.objects.filter(pub_date__range=(start_date, end_date))

6、year： one of the years

Entry.objects.filter(pub_date__year=2005)

7、month：one of the months

Entry.objects.filter(pub_date__month=12)

8、day：a day

Entry.objects.filter(pub_date__day=3)

9、week_day： a weekday

Entry.objects.filter(pub_date__week_day=2)

source: https://www.cnblogs.com/linjiqin/p/3821914.html

If you need to use get.all, then just not use "order_by_time", in normal situation, not use order_by_time can make your requset faster, you just need to order it after get these datas.

Answer 3

It's not a problem of the serializer, the problem its on the query.

You are getting all the objects, I assume there is a big number of them because of the pagination and also you want them all ordered by time. The problem is that the Model definition has no hint for the database to create a index on the time field.

Try to add the hint to create an index in the database and the speed will rise.

  class MyObj(models.Model):
      id = models.BigIntegerField(primary_key=True)
      time = models.DateTimeField(db_index=True)
      type = models.CharField(max_length=5)
      user = models.ForeignKey('User', related_name='myobj')