2

I am trying to dockerize my Luigi tasks and want to run Luigid service and tasks inside a docker however I am unable to start luigid within a docker due to which my Luigi tasks do not run since it keeps trying to connect to the central scheduler.

I have created a docker file, Luigi config file and Luigid.sh file.

The docker file is as follows:

FROM python:3
COPY . /
WORKDIR /
COPY luigi.cfg luigi.cfg
RUN pip install -r requirements.txt
#RUN chmod 644 ./luigi_app.py
ENTRYPOINT ["python"]
RUN mkdir /usr/local/luigi
EXPOSE 8082
CMD ["luigid"]
CMD ["./luigi_app.py"]

EDIT: Here is another version of the docker file. 

ARG PYTHON_VERSION=3.7
FROM python:$PYTHON_VERSION-alpine3.10

ARG LUIGI_VERSION=2.8.9
ENV LUIGI_VERSION ${LUIGI_VERSION}
ENV PYTHON_VERSION ${PYTHON_VERSION}
ENV LUIGI_CONFIG_DIR /etc/luigi/
ENV LUIGI_CONFIG_PATH /etc/luigi/luigi.conf
ENV LUIGI_STATE_DIR /luigi/state

COPY . /pipeline
WORKDIR /pipeline

RUN apk add --no-cache --virtual .build-deps \
      build-base \
      gcc \
      musl-dev \
      libc-dev \
      libffi-dev \
      python3-dev \
      py-mysqldb \
     && \
      python3 -m pip install \
      luigi==${LUIGI_VERSION} \
      sqlalchemy \
      mysql-connector-python \
      pandas \
      numpy \
      request \
      && \
    apk --purge del .build-deps && \
    mkdir -p ${LUIGI_CONFIG_DIR} && \
    mkdir -p ${LUIGI_STATE_DIR}

COPY logging.cfg ${LUIGI_CONFIG_DIR}
COPY luigi.cfg ${LUIGI_CONFIG_DIR}
VOLUME ["${LUIGI_CONFIG_DIR}", "${LUIGI_STATE_DIR}"]

EXPOSE 8082/TCP

COPY luigid.sh /bin/run
RUN chmod +x /bin/run
#ENTRYPOINT ["/bin/run"]
CMD ["./luigi_app.py"]

The luigi.cfg file is 

# 1 to 1 copy - this should be a base image
[core]
default-scheduler-host = luigid-service
default-scheduler-port = 8082
default-scheduler-url  = http://luigid-service:8082/
rpc-connect-timeout    = 5
rpc-retry-attempts     = 3
rpc-retry-wait         = 30

[scheduler]
record_task_history = True
state_path = /usr/local/luigi/luigi-state.pickle
# number of seconds (aka 6 hours) tasks are kept in the UI
remove_delay = 2160

[task_history]
db_connection = sqlite://///usr/local/luigi/luigi-task-hist.db

[retcode]
# The following return codes are the recommended exit codes for Luigi
# They are in increasing level of severity (for most applications)
already_running=0
missing_data=0
not_run=0
task_failed=30
scheduling_error=35
unhandled_exception=40

[execution_summary]
# display all task after a pipeline is run and print their status
summary-length=0

EDIT Here is another version of the luigi.cfg

core]
logging_conf_file: /etc/luigi/logging.cfg

[scheduler]
record_task_history: True
state-path: /luigi/state/luigi-state.pickle

[task_history]
db_connection: sqlite:////luigi/state/luigi-task-history.db

The luigid.sh file is 

#!/bin/sh
cat << "EOF"
 _____       __    __    _____      _____     _____
(_   _)      ) )  ( (   (_   _)    / ___ \   (_   _)
  | |       ( (    ) )    | |     / /   \_)    | |
  | |        ) )  ( (     | |    ( (  ____     | |
  | |   __  ( (    ) )    | |    ( ( (__  )    | |
__| |___) )  ) \__/ (    _| |__   \ \__/ /    _| |__
\________/   \______/   /_____(    \____/    /_____(
EOF
echo "Luigi: $LUIGI_VERSION - Python: $(python --version)"

exec luigid

When I build up my docker image and run it, it gives error that it is unable to connect to the central scheduler which is basically luigid. Can someone please help me correct my files or method?

SarwatFatimaM
  • 315
  • 1
  • 3
  • 13
  • Is it the case that you (a) don't want to run luigid in the docker, (b) want to run luigid in docker, but can't, (c) are running luigid outside of docker, but can't see it from the docker container, or (d) none of the above? – MattMcKnight Sep 16 '19 at 14:34
  • It's (b). I am trying to run luigi task within docker container for which I need luigid to run. But I am unable to figure out how to do that. I am also not able to figure out using docker through luigi task. – SarwatFatimaM Sep 17 '19 at 06:47
  • You are not actually running `/bin/run` from the second `Dockerfile`; how would the service come up if you don't start it? – tripleee Sep 17 '19 at 07:00
  • I shifted the code from docker file to within python file as: ```#!/bin/sh import subprocess subprocess.Popen(['luigid', '/bin/run/luigi.sh']) ``` but this gives me error that exec command is not right format in luigi.sh file. – SarwatFatimaM Sep 17 '19 at 07:10

0 Answers0