22

I am pretty new to celery and django in general so please excuse my lack of knowledge. I am trying to run a test to do some calculations and wait for the test to finish so that I can make sure it is done for the correct answers.

Here is what i have:

In app/tests.py

from tasks import *


c = calculate.apply_async(args=[1])

# wait until the task is done
while not calculate.AsyncResult(c.id).status == "SUCCESS":
    print c.state
    pass

in app/tasks.py

from celery import shared_task

@shared_task
def calculate(proj_id):

    #some calculations followed by a save of the object

The state never changes from pending even though in the celery log it says that the task was completed successfully

[2014-06-10 17:55:11,417: INFO/MainProcess] Received task: app.tasks.calculate[1f11e7ab-0add-42df-beac-3d94c6868aac]
[2014-06-10 17:55:11,505: INFO/MainProcess] Task app.tasks.calculate[1f11e7ab-0add-42df-beac-3d94c6868aac] succeeded in 0.0864518239978s: None

I have also put CELERY_IGNORE_RESULT = False in the mainapp/settings.py, but this did not seem to do anything.

DoctorWizard
  • 303
  • 1
  • 3
  • 11
  • Test code you have is correct, problem has to be somewhere in the settings or the overall setup I presume, where exactly I can't tell without more info. – lehins Jun 11 '14 at 10:23
  • 1
    @AlexeyKuleshevich Here are the settings I am aware of, if I miss something please let me know. celery.py and __init_.py follow the tutorial conventions and here is what I have in mainapp/settings.py: `BROKER_URL = "amqp://the ipadress" CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend' CELERY_ACCEPT_CONTENT = ['json','pickle','msgpack','yaml'] CELERY_IGNORE_RESULT = False` – DoctorWizard Jun 11 '14 at 19:15
  • @AlexeyKuleshevich As you have answered my original question I have moved the other question to a more appropriate place in another post. If you wish to continue helping you can find it [here](http://stackoverflow.com/questions/24271849/celery-result-error-args-must-be-a-list-or-tuple) – DoctorWizard Jun 17 '14 at 19:31

3 Answers3

15

Straight from doc: Result backend does not work or tasks are always in PENDING state.

All tasks are PENDING by default, so the state would have been better named "unknown". Celery does not update any state when a task is sent, and any task with no history is assumed to be pending (you know the task id after all).

  1. Make sure that the task does not have ignore_result enabled.

    Enabling this option will force the worker to skip updating states.

  2. Make sure the CELERY_IGNORE_RESULT setting is not enabled.

  3. Make sure that you do not have any old workers still running.

    It’s easy to start multiple workers by accident, so make sure that the previous worker is properly shutdown before you start a new one.

    An old worker that is not configured with the expected result backend may be running and is hijacking the tasks.

    The –pidfile argument can be set to an absolute path to make sure this doesn’t happen.

  4. Make sure the client is configured with the right backend.

If for some reason the client is configured to use a different backend than the worker, you will not be able to receive the result, so make sure the backend is correct by inspecting it:

>>> result = task.delay(…)
>>> print(result.backend)
Jithin Pavithran
  • 1,250
  • 2
  • 16
  • 41
noooooooob
  • 1,872
  • 3
  • 21
  • 27
8

So, your settings are wrong. :) You also need to setup a broker for celery to work.

First of all, djcelery is deprecated, everything is included in celery for it to work with django.

Second of all, do not set all content to be accepted, it can be a potential security risk. Use pickle only in case that simple json is not enough (let's say you pass functions or objects as arguments to tasks, or return from tasks)

So my guess is, you are just trying celery out, that's why you are trying to use database backend, which is fine, but for production use I would recommend using RabbitMQ.

In any case, give it a try with those settings:

BROKER_URL = 'django://'
INSTALLED_APPS = (
    ...
    'kombu.transport.django', 
    ...
)
CELERY_RESULT_BACKEND = 'db+scheme://user:password@host:port/dbname' 
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_IGNORE_RESULT = False # this is less important

then run python manage.py syncdb

Just to let you know, I have not used database as a broker or result backend, hence setup might be incomplete, or even incorrect, but give it a try anyways.

more CELERY_RESULT_BACKEND setting for database examples

In case you want to setup RabbitMQ as a broker backend, which I would recommend and I know for sure it will work:

if on ubuntu run:

sudo apt-get install rabbitmq-server
sudo rabbitmqctl add_user <username> <password>
sudo rabbitmqctl add_vhost <vhost, use project name for example>
sudo rabbitmqctl set_permissions -p <vhost> <username"> ".*" ".*" ".*"

Then configure celery in settings.py:

BROKER_URL = 'amqp://<user>:<password>@localhost:5672/<vhost>'
CELERY_TIMEZONE = TIME_ZONE
CELERY_RESULT_BACKEND = 'amqp'
# thats where celery will store scheduled tasks in case you restart the broker:
CELERYD_STATE_DB = "/full/path/data/celery_worker_state" 
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

Let me know how it goes.

lehins
  • 9,642
  • 2
  • 35
  • 49
  • it no longer pends forever, but it is now failing. The error is telling me that I need task args to be a list or a tuple though when I make it a list or tuple it tells me it needs to be an int(). So very confusing. – DoctorWizard Jun 13 '14 at 16:03
  • Well, that's good, that means celery is working. Post your task code and error, I'll tell you where the problem is coming from. – lehins Jun 14 '14 at 05:41
  • Also you can use `c = calculate.delay(1, foo=2)` instead of `c = calculate.apply_async(args=[1], kwargs={'foo':2})` if you just want to put it on queue instead of scheduling at specific time or need to specify any custom behavior. – lehins Jun 14 '14 at 05:44
  • I can't post the code in its entirety but I can post some of it. `@shared_task def calculate(project_id): project = Project.objects.get(project=project_id) for part in project.part_set.all(): partCalculate(part.id) p = Part.objects.get(id=part.id) #add various things to the project project.save() def partCalculate(part_id): part = Part.objects.get(id=part_id) #initalize various lists for p in part.subpart_set.all(): # put stuff in lists if any(of the values above): #some statistics stuff part.save()` – DoctorWizard Jun 16 '14 at 17:08
  • `[INFO/MainProcess] Received task: measuring.tasks.calculate[id] [ERROR/MainProcess] Task measuring.tasks.calculate[id] raised unexpected: ValueError('task args must be a list or tuple',) Traceback: File "/celery/app/trace.py", line 240, in trace_task R = retval = fun(*args, **kwargs) File "/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs) File "/calculate/tasks.py", line 28, in calculate p = Part.objects.get(id=part.id) File "/python2.7/site-packages/celery/app/task.py", line 555, in apply_async` – DoctorWizard Jun 16 '14 at 17:21
  • Sorry, formatting correctly in comments is very difficult `**dict(self._get_exec_options(), **options) File "/python2.7/site-packages/celery/app/base.py", line 351, in send_task reply_to=reply_to or self.oid, **options File "celery/app/amqp.py", line 252, in publish_task raise ValueError('task args must be a list or tuple') ValueError: task args must be a list or tuple` – DoctorWizard Jun 16 '14 at 17:23
  • I am glad you moved it to a new issue, it's impossible read code in comments :) – lehins Jun 18 '14 at 21:41
0

If you are using old django-celery and RabbitMQ as result backend, then these settings may help:

# Mostly, all settings are the same as in other answers

CELERY_RESULT_BACKEND = 'rpc://'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_IGNORE_RESULT = False

# This line is what I needed
CELERY_TRACK_STARTED = True
Denis Krumko
  • 301
  • 3
  • 5