0

I am looking to run a dockerized python script once every minute and I would like to understand how to make this script run every minute and would I do it via cron?

It will be collecting data via an api call and loading to a mysql db in another docker container I have set up, but for now as an example here is what I have running:

Dockerfile:

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "python", "./loading_data.py" ]

loading_data.py:

print('i am loading data to mysql...')

I have my docker python image built:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
docker_python       latest              f7222d16bbce        3 minutes ago       938MB

If I run the image it will execute the printing statement upon being built correct? So do I want to:

  1. write into the python script an infinite loop and have my OS start the docker container and restart it if it fails? (feels wrong)

  2. do I want to have a cron job start the container, destroy it and restart it every minute? (feels excessive)

  3. Or is there a simpler more elegant and better way?

--------------------------------------------------Attempt:

I've been trying to setup the python script as a cron job as suggested following https://github.com/cheyer/docker-cron example here is my build:

Dockerfile:

 #python install----------
    FROM python:3.6 
    WORKDIR /usr/src/app
    COPY requirements.txt ./
    RUN pip install --no-cache-dir -r requirements.txt
    COPY . .
    CMD [ "python", "./loading_data.py" ]
    #cron install----------
    FROM ubuntu:latest
    # Install cron
    RUN apt-get update
    RUN apt-get install cron
    # Add crontab file in the cron directory
    ADD crontab /etc/cron.d/simple-cron
    # Add shell script and grant execution rights
    ADD script.sh /script.sh
    RUN chmod +x /script.sh
    # Give execution rights on the cron job
    RUN chmod 0644 /etc/cron.d/simple-cron
    # Create the log file to be able to run tail
    RUN touch /var/log/cron.log
    # Run the command on container startup
    CMD cron && tail -f /var/log/cron.log

script.sh:

python ./loading_data.py >> /var/log/cron.log 2>&1

crontab:

* * * * * root /script.sh
# An empty line is required at the end of this file for a valid cron file.

Built and ran the container and the cron job runs no problem every minute, but python can't be found within the container. How would I go about repairing this issue? I believe I am very close:

sudo docker exec -i -t 3c54ffaf2674db1ab0751abe93ce77956d8b5b594a7c48cc589c4841761d4e71 /bin/bash 
    root@3c54ffaf2674:/# cat /var/log/cron.log
    /script.sh: 1: /script.sh: python: not found
    /script.sh: 1: /script.sh: python: not found
    /script.sh: 1: /script.sh: python: not found
LoF10
  • 1,907
  • 1
  • 23
  • 64

1 Answers1

0

As you already listed down the solutions and their pros and cons. 1 way in which I would be doing is, write a corn job which will run the script "python ./loading_data.py" every minute and place it into the same container which means container is always running and it will run the cornjob as well inside.

nischay goyal
  • 3,206
  • 12
  • 23
  • 1
    So I am not 100% sure I understand. As far as I know, to write a cron job I would normally run crontab -e and write in the details to execute python ./loading_data.py. Are you suggesting that I install cron in the same docker container as the python script? I see a post here on running cron jobs inside a container. Appreciate the help: https://stackoverflow.com/questions/37458287/how-to-run-a-cron-job-inside-a-docker-container – LoF10 Mar 29 '20 at 18:21
  • 1
    Yeah, Install the crontab in the same container. – nischay goyal Mar 29 '20 at 18:25
  • Hey Nischay, I added my attempt using what I believe to be your recommendations, would appreciate if you can take a look and let me know what you think? – LoF10 Mar 30 '20 at 13:41