2

I am using this docker-compose.yaml file to run airflow on docker container.

https://airflow.apache.org/docs/apache-airflow/2.0.2/docker-compose.yaml

I need to install JRE in one of the containers. How do I add instruction to add java to the docker-compose.yaml file?

Venkat
  • 51
  • 1
  • 5

2 Answers2

4

Try the following:

Create the following Dockerfile in the directory where you also have the docker-compose.yml:

FROM apache/airflow:2.0.2

USER root

# Install OpenJDK-11
RUN apt update && \
    apt-get install -y openjdk-11-jdk && \
    apt-get install -y ant && \
    apt-get clean;

# Set JAVA_HOME
ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64/
RUN export JAVA_HOME

USER airflow

WORKDIR /app

COPY requirements.txt /app

RUN pip install --trusted-host pypi.python.org -r requirements.txt

In order to install packages via apt, you have to switch to root user. Later, we switch back to user airflow. The specification of the working directory and the installation of python packages through requirements.txt might not apply, depending on your case.

Then, in your docker-compose.yml, add build: . after &airflow-common.

Finally, build your pipeline using docker-compose up -d --build.

For more information, look here: https://airflow.apache.org/docs/docker-stack/build.html#building-the-image

Requin
  • 467
  • 4
  • 16
0

You have to build your own airflow image. In your Dockerfile you have to apt install the airflow and export JAVA_HOME.

Dockerfile example:

FROM apache/airflow:2.5.1
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         openjdk-11-jre-headless \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
USER airflow
ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
RUN pip install --no-cache-dir apache-airflow-providers-apache-spark==2.1.3

Build command example:

docker build -t airflow-with-java .

After successfully image building edit your docker-compose by replacing airflow image:


x-airflow-common:
  &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-airflow-with-java}

More information you can look here: https://airflow.apache.org/docs/docker-stack/build.html

Leandro Queiroz dos Santos
  • 16
  • 1