10

I want to use Celery to run jobs on a GPU server with four Tesla cards. I run the Celery worker with a pool of four workers such that each card always runs one job.

My problem is how to instruct the workers to each claim one GPU. Currently I rely on the assumption that the worker processes should all have contiguous process IDs:

device_id = os.getpid() % self.ndevices

However, I this is not guaranteed to always work, i.e. when worker processes get restarted over time. So ideally, I would like to get the ID of each worker directly. Can someone tell me if it is possible to inspect the worker from within a task or can suggest a different solution to distribute the jobs across the GPUs?

oceanhug
  • 1,352
  • 1
  • 11
  • 24

1 Answers1

15

If you are using CELERYD_POOL = 'processes', the worker pool is handled by billiard, which does happen to expose its 0-based process index:

from billiard import current_process
from celery import task

@task
def print_info():
    # This will print an int in [0..concurrency[
    print current_process().index

The index is 0-based, and if a worker happens to be restarted it will keep its index.

I couldn't find any documentation regarding the index value though :/

Nicolas Cortot
  • 6,591
  • 34
  • 44
  • 1
    I think I was unclear with my terminology: I actually have one worker which spawns four processes. What I'm looking to get is the process ID within Celery (0-3), rather than the UNIX pid. I guess having multiple Celery workers with just one process each would work but would be less convenient. It does not really matter where I have access to the ID: it could either be inspected at the task level or be a global variable for each process. – oceanhug Jun 18 '13 at 21:44
  • 1
    Agreed, I'm not very satisfied with my solution either... It's OK if you have a production deployment with some workers managed with supervisord, but it requires a dedicated configuration and won't consume available ressources automatically based on a single `--concurrency=xxx` argument. There *is* a `current_app.pool`, but I cant'figure out what to do with it... – Nicolas Cortot Jun 18 '13 at 21:54
  • 1
    Edit: found a whole new answer \o/ – Nicolas Cortot Jun 18 '13 at 23:10
  • 1
    Care to share your new answer? :) – grim Feb 24 '15 at 19:01
  • @grim - I believe Nicholas is referring to an actual re-write of his answer. In the edit history of this response, there was a fully different answer here earlier. – Geoff Dec 06 '16 at 22:26
  • [edited comment] I'm noticing that `current_process()` may not have index. You can still get the `.pid`. This probably indicates that it is not a celery process, and some of my brief testing seems to validate this notion. – AlanSE Dec 13 '16 at 16:07
  • 2
    in python3 the import is ```from billiard.process import current_process``` it has index also. ```current_process().index``` works – H S Rathore Dec 31 '20 at 10:37