2

I'm using:

  • Celery 3.1.23
  • Redis 2.10.5 (as Celery backend)
  • Tornado 4.4
  • Python 2.7 (sigh)
  • macOS 10.10.5

I've seen a couple dozen similar questions on Stack Overflow, but none of them appear to be helping my situation. I tried --pool solo, and I checked my config for enable_result_backend=True and CELERY_IGNORE_RESULT=False, and they appear to be fine.

I'm doing a REST API served by Tornado, which has a timeout of 10 seconds.

In that Tornado endpoint, I create a Celery task with a known (derived) task_id. This task_id is derived from the Tornado inputs, so it's always the same for the same request. The idea is to have a single "get this result" function with a single URL that can be used both to initiate a request and to pick up the result of the request once it's done.

Anyway, I then wait a while for the celery task to finish - but it doesn't always finish before Tornado times out, and shouldn't be expected to either (I'm adding an artificial sleep for the sake of testing that).

When the Tornado endpoint times out, the client of the endpoint reconnects with the same inputs, and I re-derive the (same) task_id, and attempt to get the already-running task's status.

However, that status is always PENDING. I can actually collect the result of the previous celery task by recreating another Celery task with the same task_id. It's as though I'm getting 2 Celery tasks with the same task_id, and the 2nd one lets me see the result of the 1st. So things seem like they are working fine to the caller, but I actually get the same celery task running twice (or more).

I'm first checking for the task, to see if it already exists, like the following. Note that this is NOT how the Celery FAQ says to do it:

async_result = celery.result.AsyncResult(id=my_uuid)

If that async_result.status comes back as 'PENDING', then I create the task like:

async_result = ZipUp.apply_async(args=args, task_id=my_uuid)

The problem seems to be that the first async_result is always PENDING, whether I wait long enough for the task to finish or not.

What am I missing?

I repeat: I checked a couple dozen Stack Overflow questions, and found some things that looked similar, but none of them appear to be helping.

Thanks!

dstromberg
  • 6,954
  • 1
  • 26
  • 27
  • I just tried CELERY_TRACK_STARTED=True, but it doesn't appear to be helping. – dstromberg Aug 09 '16 at 21:17
  • I just saw the following in the Celery FAQ: Can I use natural task ids? Answer: Yes, but make sure it is unique, as the behavior for two tasks existing with the same id is undefined. The world will probably not explode, but at the worst they can overwrite each others results. ----- So it's pretty important that I not start another task with the same ID. – dstromberg Aug 09 '16 at 22:56
  • I've [found](http://stackoverflow.com/a/38267978/1906307) that using `AsyncResult` has significant gotchas. I would suggest storing the results of computation in a way that is independent from Celery task ids. If you insist on using task ids, then you should edit the question with a [mcve] because the devil is in the details. – Louis Aug 10 '16 at 10:54

0 Answers0