0

I'm running into a really vexing issue when using Celery with RabbitMQ.

I have something like:

from celery import group
group_of_tasks = group(task.s(x) for x in job_list)
result = group_of_tasks.delay()
print result.id  # let's call this d453359d...

The above works fine, no issues, I can query the states of the group as well as the individual AsyncResults in result.results.

However, if I try to, in a separate console, do the following:

from celery.result import GroupResult
x = GroupResult.restore('d453359d...')

I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\celery\result.py", line 806, in restore
     ).restore_group(id)
  File "C:\Python27\lib\site-packages\celery\backends\amqp.py", line 297, in restore_group
    'restore_group is not supported by this backend.')
NotImplementedError: restore_group is not supported by this backend.

I came across a similar question in Retrieving GroupResult from taskset_id in Celery?, where the .save is mentioned, but using that also results in a NotImplementedError being thrown.

The source for the AMQP backend defined in celery.backends.amqp has the following:

def save_group(self, group_id, result):
    raise NotImplementedError(
        'save_group is not supported by this backend.')

def restore_group(self, group_id, cache=True):
    raise NotImplementedError(
        'restore_group is not supported by this backend.')

So, my question is, is there no way for me to recreate a group of results using just the GroupResult ID? The only way to do it would be to store the IDs of each of the AsyncResults in the group and query each one?

Or am I just missing something very obvious here?

I'm running Celery on Python 2.7.10 on Windows, using RabbitMQ.

Community
  • 1
  • 1
DefPlayr
  • 295
  • 1
  • 3
  • 6

1 Answers1

1

You are correct - you cannot recreate the group of results while using RabbitMQ - you will need to use a different results backend that does support this action, such as Redis.

scytale
  • 12,346
  • 3
  • 32
  • 46
  • Thanks for the answer! I had a feeling there was no way with AMQP (I'll have to stick with that because we need reliability and persistence), but wanted to make sure I wasn't missing something. – DefPlayr Nov 23 '15 at 18:15
  • 2
    you do realise that redis offers [persistence](http://redis.io/topics/persistence) and [replication](http://redis.io/topics/replication), right? – scytale Nov 24 '15 at 10:57
  • Wow. This is why I love SO. Thanks for those links, @scytale, I'll take a look and see if Redis would make more sense for this project. – DefPlayr Nov 24 '15 at 18:36
  • 1
    I'd recommend rabbitmq for the broker and redis for the results backend – scytale Nov 25 '15 at 10:07