Simple query causes memory leak in Django

Question

I work in a company that has a large database and I want to perform some update queries on it but it seems to cause a huge memory leak the query is as follow

c= CallLog.objects.all()
for i in c:
   i.cdate = pytz.utc.localize(datetime.datetime.strptime(i.fixed_date, "%y-%m-%d %H:%M"))
   i.save()

I wrote this in the interactive shell of Django

I even tried to use

with transaction.atomic()

but it didn't work, do you have any idea how can I detect the source of

the dataset I am working on is about 27 million

fixed_date is a calculated property

Why do you say that it is a memory leak? What is happening when you run this code? Any error stacktrace that you could add? — Ralf, Apr 04 '19 at 18:08
i have a 4 gb memory machine and when i run htop , i see all the memory got eaten by Django and the system freeze after that — Ahmed Ibrahim, Apr 04 '19 at 18:22
You need to use Redis or some kind of Queue handler to process all the data 1 by 1 — Eddwin Paz, Apr 04 '19 at 19:58

mathias.lantean · Accepted Answer · 2019-04-04T20:06:33.443

1

You could try something like this:

from django.core.paginator import Paginator

p = Paginator(CallLog.objects.all().only('cdate'), 2000)
for page in range(1, p.num_pages + 1):
    for i in p.page(page).object_list:
        i.cdate = pytz.utc.localize(datetime.datetime.strptime(i.fixed_date, "%y-%m-%d %H:%M"))
        i.save()

Slicing a query set does not load all the objects in memory only to get a subset but adds limit and offset to the SQL query before hitting the database.

edited Apr 04 '19 at 20:06

answered Apr 04 '19 at 18:23

mathias.lantean

541
4
10

i think this piece of code update the whole table with the same value, and i want to update every record with it's calculated fixed_date value. i thought about F() expression but it doesn't work with calculated properties – Ahmed Ibrahim Apr 04 '19 at 18:27
@AhmedIbrahim I edited my answer. Try this, it should work a little better. If this takes long time and is memory consuming you should think on create an asynchronous task and let the user know when it's done. – mathias.lantean Apr 05 '19 at 10:29
I'll try this one and get back to you – Ahmed Ibrahim Apr 07 '19 at 16:29
it works perfectly but so slow , but it doesn't matter as i am doing this one time only – Ahmed Ibrahim Apr 08 '19 at 09:51

score 0 · Answer 2 · answered Apr 04 '19 at 18:41

You could try to iterate the queryset in batches; see the .iterator() method. See if that improves anything

for obj in CallLog.objects.all():
    obj.cdate = pytz.utc.localize(
        datetime.datetime.strptime(obj.fixed_date, "%y-%m-%d %H:%M"))
    obj.save()

Here is a related answer I found, but it is a few years old.

score 0 · Answer 3 · answered Apr 04 '19 at 18:47

0

Try breaking it into small blocks (since you have only 4gb of ram)

c= CallLog.objects.filter(somefield=somevalue)

When its necessary, I usually use a character or number (ID enting in 1,2,3,4 etc)

answered Apr 04 '19 at 18:47

Walucas

2,549
1
21
44

Simple query causes memory leak in Django

3 Answers3