No BS4 Module in Docker Airflow

Question

Basically How can I install pip file in docker for a complete beginner. Error says no BS4 module installed and I Don't figure it out how to install pip file and then build it.

Answer 1

You have to build your custom image. The Docker Image of Airflow contains most common used set of dependencies. It uses a set of basic "extras" that are needed there. But if you want to install dependencies for any "serious" use you have to either "Extend" (easier) or "Customize" the image (more complex).

In your case likely "extending" image is what you need.

The ready-to-use recipes on how to do either of those, including examples can be found in pretty comprehensive documentation we have in Airflow:

https://airflow.apache.org/docs/docker-stack/index.html

Some more direct links:

Building image - with explanation of whys and when's and instructions on how to build your own image: https://airflow.apache.org/docs/docker-stack/build.html

Example of adding new package (something that you need)

https://airflow.apache.org/docs/docker-stack/build.html#adding-a-new-pypi-package

Also for quick development and testing we have the option of installing packages "on-the-flight" when you start the image. This is very bad for production use, but for quick testing might be good enough:

https://airflow.apache.org/docs/docker-stack/entrypoint.html#installing-additional-requirements

It can be utilised in test/development set-up of docker compose for example:

https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html

But when you test it for such development testing you should build your own image eventually.