3

In my celery tasks, I have a task which uses a python module(theano) that runs on GPU, this module can only be imported by one thread at the same time. But to start the website, I have to run:

python manage.py runserver
celery -A celery_try worker -l info

So the module will be imported by both celery and django website, which is conflict. Is there a way to decouple Django and Celery so that the module is only imported once?

Demonedge
  • 1,363
  • 4
  • 18
  • 33
  • I'm not sure why running both processes means you have to import the module in both. If you don't want to import it in one of them, just don't. – Daniel Roseman May 13 '16 at 08:55
  • In Django view, I import the tasks, so Django server imports the same modules tasks.py imports, while celery imports whatever tasks.py imports too, that's how both process imports the module. – Demonedge May 13 '16 at 08:59
  • celery itself is runing outside of django by default! it is how you initialize it matter, e.g. if your module is loading all django stack and all your django apps – Jerzyk Jun 05 '16 at 10:39
  • one more thing - each worker will be running in its own subprocess or thread - so if you have 2 workers each one will load theano module and you will have same issue, maybe write your own minimal server to handle tasks, you wil be sure that is runing once and you have full control on how it is running – Jerzyk Jun 05 '16 at 10:43

1 Answers1

0

For testing purposes you can run django development server in single-threaded mode: python manage.py runserver --nothreading.

You want to import theano only in celery worker process, not in django web server process, right? Ok, let's make import conditional, so that it is imported in celery and not in django. 

import os

try:
    # next line will raise exception in django, but will work fine in celery
    is_worker = os.environ['celery_worker']  
    import theano  # celery will import theano, django won't
except Exception as exc:
    # django code will catch exception that celery_worker doesn't exist and print it here
    print exc

And start your celery worker with celery_worker environment variable set:

celery_worker=yes celery -A celery_try worker -l info

To discriminate between celery worker and django, let's set a bash environment variable in celery process, but not in django process. I called that variable celery_worker. In order to set it, I prepended celery -A celery_try worker -l info with per-command env variable assignment: celery_worker=yes. Now, in python code, I check if the environment variable is present. If it is, we are in celery worker and need to import theano.

If we're in django, os.environ['celery_worker'] shouldn't be defined and should raise an exception.

Community
  • 1
  • 1
Boris Burkov
  • 13,420
  • 17
  • 74
  • 109
  • `python manage.py runserver --nothreading` doesn't work, I still get the same error message. And I don't quite understand your decoupling solution, could you pls explain more? Thanks. – Demonedge May 13 '16 at 09:07
  • @Demonedge You want to `import theano` only in celery worker process, not in django web server process, right? Ok, let's make import conditional, so that it is imported in celery and not in django. To discriminate between celery worker and django, let's set a bash environment variable in celery process, but not in django process. I called that variable `celery_worker`. In order to set it, I prepended `celery -A celery_try worker -l info` with per-command env variable assignment: `celery_worker=yes`. Now, in python code, I check if the environment variable is present and if it is import theano. – Boris Burkov May 13 '16 at 09:33
  • The problem is that I can't choose not to import Theano. When I start the server, views.py will import tasks from Celery automatically, which will import Theano. When I run Celery work to process the task, again Celery will import tasks.py, which will again import Theano. – Demonedge May 16 '16 at 05:10
  • No, you can! Just make the import conditional. I added some in-code comments to my example, so that it's more explicit. If `os.environ['celery_worker']` does not exist, the theano won't be imported, if it exists - theano will be imported. If you run `python manage.py runserver`, celery_worker is undefined and theano isn't imported. If you're running celery, celery_worker is defined, thus import happens. – Boris Burkov May 16 '16 at 10:34