Using Docker with Python Poetry?

Question

I've been using Docker with pipenv for my data science deployment setup and now I want to change to Poetry. My Dockerfile is:

FROM python:3.8-alpine3.13

ENV POETRY_VIRTUALENVS_CREATE=false \
    POETRY_VERSION=1.1.11

RUN apk add --no-cache python3-dev gcc libc-dev musl-dev openblas gfortran build-base postgresql-libs postgresql-dev libffi-dev \
  && pip install poetry
COPY pyproject.toml poetry.lock ./
RUN poetry export -f requirements.txt --output requirements.txt && sed -i 's/^-e //' requirements.txt

USER root
RUN apt-get update && apt-get install -y --no-install-recommends python3-dev gcc libc-dev musl-dev openssh-client git libpq-dev \
  && apt-get clean -y


# install dependencies from requirements.txt
RUN pip install --no-cache-dir --user -r requirements.txt

COPY . .
CMD ["jupyter", "notebook", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]

My pyproject.toml:

[tool.poetry]
name = ""
version = "1.0.0"
description = ""
authors = [""]

[tool.poetry.dependencies]
python = "^3.8"

... # lots of libraries, omitted here

[tool.poetry.dev-dependencies]
black = "*"
ipykernel = "6.*"
ipython = "7.*"
isort = "5.*"
jupyter = "*"
pytest = "6.*"
pre-commit = "2.*"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

This is based on other Dockerfiles that I found on StackOverflow. I run into the following problem:

Step 7/10 : RUN apt-get update && apt-get install -y --no-install-recommends python3-dev gcc libc-dev musl-dev openssh-client git libpq-dev   && apt-get clean -y
 ---> Running in 447ffb83d555
/bin/sh: apt-get: not found
The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends python3-dev gcc libc-dev musl-dev openssh-client git libpq-dev   && apt-get clean -y' returned a non-zero code: 127
Running containerdocker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "jupyter": executable file not found in $PATH: unknown.
ERRO[0000] error waiting for container: context canceled 
make: *** [Makefile:6: jupyter_notebook] Error 127

So this looks like the Poetry is not used, where Jupyter is installed, and therefore it cannot be found. How can I fix this?

@OneCricketeer because they contain a lot of packages and I don't need about half of them — qalis, Dec 07 '21 at 09:16

score 5 · Answer 1 · answered Dec 06 '21 at 13:47

It is not a problem with poetry, but because you're trying to install packages using apt-get in an alpine image (which doesn't include apt-get). You're also installing some of the packages multiple times, since you're using first using apk and then using apt-get. Also you don't need to install python, since you're using the python image as base.

I would also suggest just using poetry to install instead of dumping to requirements.txt and installing poetry using the official installer. You will also have to bump python since python 3.8 is not supported.

Take a look at this answer for combining docker and Poetry: Integrating Python Poetry with Docker

Your Dockerfile could look like this instead:

FROM python:3.9-alpine3.13

ENV POETRY_VIRTUALENVS_CREATE=false \
    POETRY_VERSION=1.1.11 \
    YOUR_ENV=development

RUN apk add --no-cache python3-dev gcc libc-dev musl-dev openblas gfortran build-base postgresql-libs postgresql-dev libffi-dev curl

RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -
ENV PATH "/root/.local/bin:$PATH"


# install dependencies
COPY pyproject.toml poetry.lock ./
RUN poetry install $(test "$YOUR_ENV" == production && echo "--no-dev") --no-interaction --no-ansi

COPY . .
CMD ["jupyter", "notebook", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]

Thanks for many helpful suggestions. However, while this seems to work OK with plain Jupyter, I run into 2 problems while adding actual packages (e.g. NumPy, PyTorch). First, installing takes forever compared to pipenv, which is the exact opposite of what I wanted to achieve, about 15 m instead of less than 5 minutes (and exact opposite of what I witnessed in other projects). Secondly, I get errors and it just does not work. Full stack trace: https://pastebin.com/5HSQyVvV — qalis, Dec 07 '21 at 09:47
To get rid of errors I had to use Jupyter Docker Stack after all — qalis, Dec 07 '21 at 10:05
For what its worth, you might want to try using a debian-based image like `3.9-slim`. From past experience, there are a lot python packages that don't have pre-compiled wheels for Alpine, so you end up needing to download and compile the source. My alpine images built slower and were larger than `slim` images because of that. I'd also recommend setting `POETRY_HOME` — mrosales, Dec 17 '21 at 20:31

score 1 · Accepted Answer · answered Dec 07 '21 at 10:06

Based on Asgeer's answer I managed to fix the Dockerfile. However, using Alpine turned out to be pretty impossible and the final solution is:

FROM jupyter/scipy-notebook:latest

ENV POETRY_VIRTUALENVS_CREATE=false \
    POETRY_VERSION=1.1.11

USER root
RUN apt-get update \
    && apt-get --yes install apt-utils \
    && apt-get --yes install curl

RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -
ENV PATH "/home/jovyan/.local/bin:$PATH"

# install dependencies
COPY pyproject.toml poetry.lock ./
RUN poetry install --no-interaction --no-ansi

COPY . .
CMD ["jupyter", "notebook", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]

Using Docker with Python Poetry?

2 Answers2