Scheduling expensive calculation and using precalculated values between requests in Django

Question

I have some function expensiveFunction() that takes several seconds to execute and returns a largish (5mb in JSON) array.

How can I schedule django to execute the function every x hours and then store the results somewhere so that when a user accesses a view, it returns those precalculated stored results?

Reading/writing to a textfile would have the overhead of reading in the file and then parsing JSON, and I feel like django sessions aren't suitable for this amount of data.

Using django 1.9.7 and python 3.4

There are multiple ways, so you may need to provide details to choose the best solutions: You can create a management command and crontab it, or you can make the JSON result a view of its own and cache it... — raphv, Jun 15 '16 at 14:54

score 0 · Accepted Answer · answered Jun 15 '16 at 14:54

0

You could have a scheduled task in Celery which is a Distributed Task Queue and save the result from that expensiveFunction() in Redis which is a key-value store and will reduce the overhead of reading from the file.

answered Jun 15 '16 at 14:54

kpantic

79
3

score 0 · Answer 2 · answered Jun 15 '16 at 14:57

You should install an asynchronous app to work with django and create cron processes.

Nowadays the standard is Celery You could also research about Django Channels, that will be included in Django 1.10, but at this moment works as a Django separated app.

Scheduling expensive calculation and using precalculated values between requests in Django

2 Answers2