2

How do I make the celery -A app worker command to consume only a single task and then exit.

I want to run celery workers as a kubernetes Job that finishes after handling a single task.

I'm using KEDA for autoscaling workers according to queue messages. I want to run celery workers as jobs for long running tasks, as suggested in the documentation: KEDA long running execution

WolfThreeFeet
  • 73
  • 7
  • If all you need is to execute a single task, then I see no point of running Celery worker. Simply have a script in your container that just calls the same function that would otherwise be called by Celery... – DejanLekic Jun 08 '21 at 09:44
  • I want to reuse celery infrastracture. Without it I'll need to implement producer consumer logic. Celery just makes it easier and gives alot of benefits. Also because I already use celery for small tasks with autoscaling deployments, I want the task calls to be consistent across the code. – WolfThreeFeet Jun 08 '21 at 10:06
  • I did not say you do not use celery infrastructure, just that you do not need Celery worker. – DejanLekic Jun 08 '21 at 17:01

1 Answers1

2

There's not really anything specific for this. You would have to hack in your own driver program, probably via a custom concurrency module. Are you trying to use Keda ScaledJobs or something? You would just use a ScaledObject instead.

coderanger
  • 52,400
  • 4
  • 52
  • 75
  • 2
    You need to go with the other approach listed there, ScaledObjects+Deployments+long-ass terminationGracePeriodSeconds. One of mine is 12 hours long :) It works fine, just as it says you'll see pods in Terminating for a while sometimes. Just make sure you tune your scale down cooldown time too so it doesn't happen too aggressively. – coderanger Jun 08 '21 at 05:44
  • 2
    Also make sure you have have ACKS_LATE, a low prefetch multiplier, and use the `http` mode in Keda for RabbitMQ if that's your broker (not the `amqp` mode). – coderanger Jun 08 '21 at 05:45
  • Feel free to ping me in #keda on Slack if you want more help, if the above isn't enough to make it clear, I revamped a lot of Keda specifically for Celery use cases and use a ton of it myself (hundreds of scalers). – coderanger Jun 08 '21 at 05:47
  • 1
    I'm using RabbitMQ. Out of curiosity, why use `http`? – WolfThreeFeet Jun 08 '21 at 06:05
  • 2
    The `amqp` poller mode doesn't see messages sitting in unack'd and so will be over-aggressive in scaling down. `http` does see those, so combined with ACKS_LATE and you've got a more stable data source. – coderanger Jun 08 '21 at 06:40