0

I have downloaded the latest Docker image for the Airflow and am able to spin up the instance succesfully. On my local system I have installed minio server using homebrew on my Mac.

I have created a DAG file to upload data to my Minio bucket. I have done a sample upload using python and it is working as expected (using the minio python libraries). On the Airflow server I am seeing the following errors -

ModuleNotFoundError: No module named 'minio'

Can someone pleae help me how can I have the pip3 minio library to the docker container so that this error can be resolved? I am new to containers and would really appreciate a easy guide or link that I can refer to help me with this error.

One of the things I did try is to fiddle with the attribute - _PIP_ADDITIONAL_REQUIREMENTS that comes in the AIRFLOW DOCKER image following this link but to no avail.

I added the values as - minio but didn't work.

VKarthik
  • 1,379
  • 2
  • 15
  • 30

1 Answers1

0

you can create a Dockerfile that extend the basic airflow and install your packages.

  1. Create Dockerfile
FROM apache/airflow:2.3.0

USER root

RUN apt-get update

USER airflow

RUN pip install -U pip

RUN pip install --no-cache-dir minio # or you can copy requirments.txt and install from it
  1. Build your docker

docker build -t my_docker .

  1. Run the new docker image (if you are using the docker-compose then change the airflow image to your image)
ozs
  • 3,051
  • 1
  • 10
  • 19
  • Thanks @ozs. I am getting the following error when running the steps you mentioned -failed to solve with frontend dockerfile.v0: failed to read dockerfile: error from sender: open .Trash: operation not permitted. Here is the command I am running - docker build -t airflow_minio . – VKarthik May 23 '22 at 23:25
  • @VKarthik, hi. it seems like something with the docker itself or the name of the file. see this https://stackoverflow.com/questions/64221861/an-error-failed-to-solve-with-frontend-dockerfile-v0 for solution – ozs May 24 '22 at 06:04