Run periodic tasks

Question

So I have a webpage powered by Django. I have a bunch of users in a database, and I want to run a certain task on every user in the database every 5 - 10 mins. What's the best way to do this? There could be up to 1000 users at a time. Is something like Celery useful for this?

Celery is good and working, but it needs a broker queue and worker instances. Use it if you really need some real-time distributed hard-core jobs to be done, don't use it if only a single scheduled job is needed — tdihp, Dec 07 '12 at 04:11
What are broker queues and worker instance. Well each user would have a bunch of webpage urls in the database. So for each user in the database, i would have to fetch each of there webpages and process them. Would Celery be best for this or would one of the ones mentioned below — kiznore, Dec 07 '12 at 04:30
About celery, please check it's official tutorial & docs. I don't think Celery is best, or even a choice, if only timely scheduled tasks should be done. Although Celery has Django binding, it will not make task like yours more easier than a cron config + simple script. If you want to access Django ORM in the script, it's not that hard, do some searching. — tdihp, Dec 07 '12 at 04:38

ptrck · Answer 1 · 2012-12-07T03:15:42.600

2

Is something like celery usefull for this

Yes, Celery comes with built in perodic tasks: Celery Beat.

Another option would be PythonRQ in combination with RQ Scheduler. This is what I've used in a recent project and I am very happy with it.

edited Dec 07 '12 at 03:15

answered Dec 07 '12 at 03:06

ptrck

731
5
13

score 1 · Answer 2 · edited May 23 '17 at 12:25

I think below is what you want

http://code.google.com/p/django-cron/

Django - Set Up A Scheduled Job?

or you just use cron on linux level

details guide for how to setup a cron :

http://www.cyberciti.biz/faq/how-do-i-add-jobs-to-cron-under-linux-or-unix-oses/

cron job in linux will run script that on predefined time interval , minutes ,hourly, daily, weekly

for details you just type "crontab -e" on your Linux terminal

crontab -e

then add up a row (task)

1 2 3 4 5 /path/to/command arg1 arg2

1: Minute (0-59) 2: Hours (0-23) 3: Day (0-31) 4: Month (0-12 [12 == December]) 5: Day of the week(0-7 [7 or 0 == sunday]) /path/to/command - Script or command name to schedule

after finish ,save and exit.

+1 for the cron job advice. Becaust it's simple and it's in fact a db manipulation job, not very application related, do don't put it in your Django app. Espatially if you want to scale and have many Django instance — tdihp, Dec 07 '12 at 04:05

score 0 · Answer 3 · answered Dec 07 '12 at 03:02

"Celery" lets Python apps (like Django) run background processes easily. It was actually originally built specifically for Django.

http://celeryproject.org/

Their example showing a simple task adding two numbers:

from celery.decorators import task

@task
def add(x, y):
    return x + y

You can execute the task in the background, or wait for it to finish:

>>> result = add.delay(8, 8)
>>> result.wait() # wait for and return the result
16

Here's info on "cron"-like scheduled tasks: http://docs.celeryproject.org/en/latest/reference/celery.schedules.html

You'll probably want to install RabbitMQ also to get it working, so it might be more complicated of a solution than you're looking for, but it will achieve your goals. There are also lighterweight ways to install it if you like (including using the DB itself) and if you understand the limitations.

Run periodic tasks

3 Answers3