8

I'm using a docker-compose. I have a web and a worker service.

version: '3'
services:
  web:
    build: .
    environment:
      - "*"
    links:
      - redis
      - memcached
    ports:
      - "80:8001"
      - "443:8001"

  worker:
    build: .
    command: ["/bin/bash", "/home/django/start_celery.sh"]
    environment:
      - "*"
    links:
      - redis
      - memcached

  memcached:
    image: memcached
    ports:
      - "11211:11211"

  redis:
    image: redis
    ports:
      - "6379:6379"

I need to run crons (scheduled tasks) on worker service.

And I dont want to hardcode the crontab in Dockerfile as I'm using same dockerfile for both the services.

So what is the best approach for this?

Roshan007
  • 658
  • 2
  • 7
  • 15
  • Possible duplicate of [How to run a cron job inside a docker container?](https://stackoverflow.com/questions/37458287/how-to-run-a-cron-job-inside-a-docker-container) – Siyu Nov 29 '18 at 11:14

4 Answers4

5

You can try the following Opensource tool for scheduling crons in the docker-compose.

https://github.com/mcuadros/ofelia

eg:

 [job-service-run "service-executed-on-new-container"]
 schedule = 0,20,40 * * * *
 image = ubuntu
 network = swarm_network
 command =  touch /tmp/example

In case you are planning to utilize the image in any of the cloud platforms.

For Eg.

AWS: You can also have a look at ECS scheduler

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduled_tasks.html

GCP: Kubernetes Engine CronScheduler

https://cloud.google.com/kubernetes-engine/docs/how-to/cronjobs

Omkar Kulkarni
  • 66
  • 2
3

Docker is supposed to run a single process on each container so a usual approach when you need cron-like tasks is to run cron on the host machine (or as a managed service like @Omkar Kulkani points) and then that "outside" launch the container running the command that you need.

something like

0,20,40 * * * * docker exec [container name] [your command here]

Carlos
  • 1,411
  • 15
  • 21
  • "is supposed to" feels fairly nebulous and the linked article is extremely long covering many topics. Any chance you could expand on the reasons you think this is the best approach? – Andy Baker Aug 17 '21 at 12:31
  • Mi firs aproach to solve this in my project was to run cron inside the docker container. That is a mistake because in this scenario two processes (at least) are running inside the container. Te solution was to put it outside and make the host machine cron to run the container with the desired command. That "is supposed to" means that even it is recommended you still can run more than one. – Carlos Sep 22 '22 at 08:25
2

We developed a simple container for scheduling jobs: amplication/scheduler it wraps an HTTP request with CRON.

Usage:

services:
  app:
     # ...
  scheduler:
    image: amplication/scheduler
    command: --request.url http://app/example-route --request.method POST --schedule '* * * * *'
Iddan Aaronsohn
  • 1,030
  • 9
  • 11
0

If you want to run scheduled task you might wanna use celery beat: https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html?highlight=periodic#starting-the-scheduler.

You can define when each task should run in cron style.

The decision where to run each task (on which worker) is done by celery router: https://docs.celeryq.dev/en/stable/userguide/routing.html

ItayB
  • 10,377
  • 9
  • 50
  • 77
  • Celery is a crazy complex dependency for something like this. It's 10s of thousands of lines of code and requires several additional long running processes. – Andy Baker Aug 17 '21 at 12:32
  • Note that both the question's tags and the docker-compose (worker service) implies that he is already using celery - that's why I suggested beat – ItayB Aug 17 '21 at 14:49
  • I didn't spot that - but... In which case that should be part of the question text. The point of SO is to answer the general question as posed for the benefit of other users. Not to infer the specific situation of the user that posted the question. – Andy Baker Aug 17 '21 at 18:18
  • he asked in the question "I need to run crons (scheduled tasks) on worker service." - that means he wants to run celery task in celery worker - this is what I tried to answer. It takes few rows to get that.. – ItayB Aug 17 '21 at 18:36
  • in which case the question title needs editing to be specific to celery. Consider me a canary in the coalmine. If I managed to miss it was about celery then so will others. – Andy Baker Aug 17 '21 at 22:35
  • That project seems to be dead? The domain doesn't work anymore... (I'm getting SSL errors at the time of writing). – Nick Mar 25 '22 at 09:36
  • 1
    @Nick seems like they changed their domain, I've updated my answer for you – ItayB Mar 25 '22 at 15:17