0

I have a task that requires ~30GiB of disk space. The Celery cluster (Redis used as broker) runs on machines with ~600GiB of disk space. I would like to somehow configure Celery so up to 10 such tasks can run on a single node. Is this possible somehow??

DejanLekic
  • 18,787
  • 4
  • 46
  • 77

1 Answers1

0

What you are asking for (task + hardware-based resource management), Celery doesn't support natively, but it shouldn't be hard to implement your own.

As for resource allocation, you can implement distributed reservations (e.g. zookeeper or redis+expiring keys), but because you're using a local filesystem, you could simply use the filesystem. For example: py-filelock, and maintain a lockfile for every resource block. It's simple; although you may need a PID based cleanup process in the event your processes are ungracefully terminated (eg kill -9, etc) and the lock is not freed. Still, relatively few lines of code required.

As for Celery, once you have your resource reservation system, you can use Reject + requeue. Use this to prevent idle workers from accepting jobs if your cannot reserve space.

Also: prefetching - you should disable this (set to 1, to avoid "hoarding" tasks by the workers): http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-worker_prefetch_multiplier

Nino Walker
  • 2,742
  • 22
  • 30
  • Now this assumes you're running heterogenous tasks on the same nodes, where some will be resource intensive, and others not. If it's a homogeneous workload of only these big tasks, you can simply configure the worker pool to exactly the number of workers to match your task/resource pool. Still, using a reservation system would allow you to handle variable sized workloads (eg add metadata to your lockfiles). – Nino Walker Jan 06 '19 at 09:40
  • 1
    so if a worker wants to reserve disk space, it updates a text file with the amount it is going to reserve so that other workers can calculate it when they are about to start their jobs? And then free up the reservation by decreasing the rerserved space amount on the file later when its done? – Forethinker Apr 27 '19 at 01:50
  • @Forethinker - exactly. You definitely want to use a filelock while manipulating these: https://stackoverflow.com/questions/489861/locking-a-file-in-python – Nino Walker May 03 '19 at 11:47