0

I'm trying to build an app with django. When I work with a small database (around 10k rows), the app works perfectly. However, in some views, I need to calculate some data of my classes, so I need to call all of my database. It is very slow (more than 30s), and according to Django Debug Toolbar, most part of the time is consumed by CPU.

Here's an example of a (very) simplified code I try to launch:

def test_view(request):
    queryset_articles = Articles.objects.all()
    total_count = sum([a.words_count for a in queryset_articles])
    response = 'Total words : ' + str(count)
    return HttpResponse(response)

How should I do to get a reasonable waiting time? I think about doing another table, which I would update frequently with these historical stats, then call them directly in my view.

Thank you,

Pierre

pythonize
  • 61
  • 1
  • 5
  • "_call all of my database_"?? That is generally not adviced and it may even _crash_ your server if there is too much data. Is the only thing you want there a _sum_ of some field? If so this question will probably give you your answer: https://stackoverflow.com/questions/8616343/django-calculate-the-sum-of-the-column-values-through-query – Abdul Aziz Barkat Jul 02 '21 at 15:41

3 Answers3

1

Try aggregation. It can provide a small to noticeable increase in time since it uses a database query to count.

from django.db.models import Sum

total_count = Articles.objects.all().aggregate(Sum('words_count'))
print(total_count)
Chillie
  • 1,356
  • 13
  • 16
1

First of all, you should check if the Django Debug Toolbar itself is not the reason for this waiting time (its documentation explains the problem and provides some solutions here).

Then, did you try to use Aggregation ? This could optimize your code:

from django.db.models import Sum

Articles.objects.aggregate(Sum('words_count'))
Mickaël Martinez
  • 1,794
  • 2
  • 7
  • 20
  • Thank you, I tried Django Toolbar after I noticed the site was very slow, so it is not the cause of the slowness – pythonize Jul 05 '21 at 13:09
0

Thank you for you answers, this view is now a lot faster! However, I got some views to calculate average with different filters, like this:

        hours_of_weekday = [Article.objects.filter(
            date__hour=k,
            date__week_day=weekday_num,
            type__in=categories).aggregate(Avg('count'))['count__avg'] for k in range(0, 24)]

Which is still very slow with this... Any idea except making a table dedicated to these values?

pythonize
  • 61
  • 1
  • 5