Dockerfile & Poetry | executor failed running [/bin/sh -c poetry install --no-dev]: exit code: 1

Question

Background

Originally, I had this post on Unix & Linux; regarding adjusting my system clock. Solved by:

sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"
sudo apt get

Then pip was failing as I had not provided credentials for an Azure DevOps download via. url. Solved by: creating a Personal Access Token and placed it into a newly established pytest.conf file.

Now, docker build command has significantly more successful installs. However, still fails.

I've not found any reference to this error on Stack Overflow; hence this post.

Poetry

Update:

Created ~/.netrc (censored):

machine pkgs.dev.azure.com
    username foo
    password bar
    save yes

pip.conf (<PAT> censored):

[global]
extra-index-url=https://<PAT>@pkgs.dev.azure.com/iotahoe/Halo/_packaging/private-sources/pypi/simple'

ls:

me@LAPTOP-G1DAPU88:~/.ssh/workers-python/workers/ontology_tagger$ ls
Dockerfile  README.md  azure-pipeline-ontology_tagger.yaml  ontology_tagger  ontology_tagger.egg-info  pip.conf  poetry.lock  pyproject.toml  pytest.ini  wss-unified-agent.config  wss_agent.sh

pytest.ini:

[pytest]
markers =
    integration: create endpoint and test returned confidence intervals
    unit: interrogate results of most recent trained model on select inputs
testpaths = tests/

Credentials Issue

Error:

:~/.ssh/workers-python/workers/ontology_tagger$ docker build --no-cache -t my_app -f /home/danielbellio/.ssh/workers-python/workers/ontology_tagger/Dockerfile .
[+] Building 1235.3s (16/23)                                                                                                                                                                               
 => [internal] load build definition from Dockerfile                                                                                                                                                  0.0s
 => => transferring dockerfile: 38B                                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/python:3.7-slim                                                                                                                                    6.9s
 => [auth] library/python:pull token for registry-1.docker.io                                                                                                                                         0.0s
 => [internal] load build context                                                                                                                                                                     0.1s
 => => transferring context: 4.48kB                                                                                                                                                                   0.0s
 => CACHED https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py                                                                                                                0.0s
 => CACHED [production 1/5] FROM docker.io/library/python:3.7-slim@sha256:798c7c77f4bba96b35de5fe9832b81084dcc4c1c52806d7976bf9233c7c3c4a6                                                            0.0s
 => [base 2/7] RUN apt-get update -y     && apt-get -y --no-install-recommends install curl wget    && rm -rf /var/lib/apt/lists/*                                                                   15.7s
 => [base 3/7] WORKDIR /home/worker/python/ontology_tagger                                                                                                                                            0.1s
 => [base 4/7] ADD https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py ./                                                                                                     0.1s 
 => [base 5/7] RUN python get-poetry.py && chmod +x /etc/poetry/bin/poetry                                                                                                                           32.3s 
 => [base 6/7] RUN --mount=type=cache,target=/root/.cache pip install twine keyring artifacts-keyring                                                                                                31.7s 
 => [base 7/7] RUN --mount=type=cache,target=/root/.cache apt update && apt install gcc -y                                                                                                           70.9s 
 => [build 1/6] COPY . .                                                                                                                                                                              0.1s 
 => [build 2/6] RUN poetry config experimental.new-installer false                                                                                                                                    3.3s 
 => ERROR [build 3/6] RUN poetry install --no-dev                                                                                                                                                  1073.6s 
------                                                                                                                                                                                                     
 > [build 3/6] RUN poetry install --no-dev:                                                                                                                                                                
#16 2.044 Creating virtualenv ontology-tagger in /home/worker/python/ontology_tagger/.venv                                                                                                                 
#16 7.182 Installing dependencies from lock file                                                                                                                                                           
#16 13.06                                                                                                                                                                                                  
#16 13.06 Package operations: 199 installs, 0 updates, 1 removal                                                                                                                                           
#16 13.06 
#16 13.06   - Removing wheel (0.37.0)
#16 14.42   - Installing six (1.16.0)
...
#16 1070.5   - Installing pwmf (0.1.3.1.dev1)
#16 1072.9 
#16 1072.9   EnvCommandError
#16 1072.9 
#16 1072.9   Command ['/home/worker/python/ontology_tagger/.venv/bin/pip', 'install', '--no-deps', '--index-url', 'https://pkgs.dev.azure.com/iotahoe/Halo/_packaging/private-sources/pypi/simple', 'pwmf==0.1.3.1.dev1'] errored with the following return code 2, and output: 
#16 1072.9   Looking in indexes: https://pkgs.dev.azure.com/iotahoe/Halo/_packaging/private-sources/pypi/simple
#16 1072.9   User for pkgs.dev.azure.com: ERROR: Exception:
#16 1072.9   Traceback (most recent call last):
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
#16 1072.9       status = self.run(options, args)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
#16 1072.9       return func(self, options, args)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 325, in run
#16 1072.9       reqs, check_supported_wheels=not options.target_dir
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
#16 1072.9       discovered_reqs.extend(self._resolve_one(requirement_set, req))
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
#16 1072.9       abstract_dist = self._get_abstract_dist_for(req_to_install)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 339, in _get_abstract_dist_for
#16 1072.9       self._populate_link(req)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 305, in _populate_link
#16 1072.9       req.link = self._find_requirement_link(req)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 270, in _find_requirement_link
#16 1072.9       best_candidate = self.finder.find_requirement(req, upgrade)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/index/package_finder.py", line 899, in find_requirement
#16 1072.9       req.name, specifier=req.specifier, hashes=hashes,
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/index/package_finder.py", line 881, in find_best_candidate
#16 1072.9       candidates = self.find_all_candidates(project_name)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/index/package_finder.py", line 826, in find_all_candidates
#16 1072.9       project_url, link_evaluator=link_evaluator,
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/index/package_finder.py", line 790, in process_project_url
#16 1072.9       html_page = self._link_collector.fetch_page(project_url)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/index/collector.py", line 643, in fetch_page
#16 1072.9       return _get_html_page(location, session=self.session)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/index/collector.py", line 455, in _get_html_page
#16 1072.9       resp = _get_html_response(url, session=session)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/index/collector.py", line 169, in _get_html_response
#16 1072.9       "Cache-Control": "max-age=0",
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py", line 543, in get
#16 1072.9       return self.request('GET', url, **kwargs)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/network/session.py", line 421, in request
#16 1072.9       return super(PipSession, self).request(method, url, *args, **kwargs)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py", line 530, in request
#16 1072.9       resp = self.send(prep, **send_kwargs)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py", line 650, in send
#16 1072.9       r = dispatch_hook('response', hooks, r, **kwargs)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_vendor/requests/hooks.py", line 31, in dispatch_hook
#16 1072.9       _hook_data = hook(hook_data, **kwargs)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/network/auth.py", line 256, in handle_401
#16 1072.9       username, password, save = self._prompt_for_password(parsed.netloc)
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/network/auth.py", line 226, in _prompt_for_password
#16 1072.9       username = ask_input("User for {}: ".format(netloc))
#16 1072.9     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 259, in ask_input
#16 1072.9       return input(message)
#16 1072.9   EOFError: EOF when reading a line
#16 1072.9   WARNING: You are using pip version 20.2.2; however, version 21.2.4 is available.
#16 1072.9   You should consider upgrading via the '/home/worker/python/ontology_tagger/.venv/bin/python -m pip install --upgrade pip' command.
#16 1072.9   
#16 1072.9 
#16 1072.9   at /etc/poetry/lib/poetry/utils/env.py:1074 in _run
#16 1073.0       1070│                 output = subprocess.check_output(
#16 1073.0       1071│                     cmd, stderr=subprocess.STDOUT, **kwargs
#16 1073.0       1072│                 )
#16 1073.0       1073│         except CalledProcessError as e:
#16 1073.0     → 1074│             raise EnvCommandError(e, input=input_)
#16 1073.0       1075│ 
#16 1073.0       1076│         return decode(output)
#16 1073.0       1077│ 
#16 1073.0       1078│     def execute(self, bin, *args, **kwargs):
------
executor failed running [/bin/sh -c poetry install --no-dev]: exit code: 1

Dockerfile (lines 5 to 7):

#https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md
#require set DOCKER_BUILDKIT=1

FROM python:3.7-slim as base
RUN apt-get update -y \
    && apt-get -y --no-install-recommends install curl wget\
    && rm -rf /var/lib/apt/lists/* 
ENV ROOT /home/worker/python/ontology_tagger
WORKDIR $ROOT

ARG ATLASSIAN_TOKEN
ARG POETRY_HTTP_BASIC_AZURE_PASSWORD
ARG ACCESS_KEY
ENV AWS_ACCESS_KEY_ID=$ACCESS_KEY
ARG SECRET_KEY
ENV AWS_SECRET_ACCESS_KEY=$SECRET_KEY
ARG REPO
ENV REPO_URL=$REPO
# Copy/symlink pypic file to the docker context before running docker build
ENV PYPIRC_PATH=$ROOT/.pypirc

ENV \
    PYTHONFAULTHANDLER=1 \
    POETRY_VERSION=1.1.4 \
    POETRY_HOME=/etc/poetry \
    XDG_CACHE_HOME=/home/worker/.cache \
    POETRY_VIRTUALENVS_IN_PROJECT=true \
    MPLCONFIGDIR=/home/worker/matplotlib \
    PATH=/home/worker/python/ontology_tagger/.venv/bin:/usr/local/bin:/etc/poetry/bin:$PATH

# System deps:
ADD https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py ./
RUN python get-poetry.py && chmod +x /etc/poetry/bin/poetry
# Copy only requirements to cache them in docker layer
RUN --mount=type=cache,target=/root/.cache pip install twine keyring artifacts-keyring
RUN --mount=type=cache,target=/root/.cache apt update && apt install gcc -y

FROM base as ws
ARG WS_APIKEY
ARG WS_PROJECTVERSION=
ARG WS_PROJECTNAME=workers-python-ontology_tagger
ARG WS_PRODUCTNAME=HALO
COPY --chown=worker:worker . .
RUN --mount=type=cache,uid=1000,target=/home/worker/.cache poetry install --no-dev
COPY --from=openjdk:15-slim-buster /usr/local/openjdk-15 /usr/local/openjdk-15
ENV JAVA_HOME /usr/local/openjdk-15
ENV PATH $JAVA_HOME/bin:$PATH
RUN --mount=type=cache,uid=1000,target=/home/worker/.cache ./wss_agent.sh

FROM base as test
COPY . .
RUN poetry config experimental.new-installer false
RUN poetry install
RUN cd ontology_tagger && poetry run invoke deploy
# RUN sphinx-build -b confluence -Q docs/source docs/build -E -a -D confluence_server_pass=$ATLASSIAN_TOKEN

FROM base as package
COPY . .
RUN poetry build
RUN python -m pip install --upgrade pip && \
pip install twine keyring artifacts-keyring && \
twine upload -r $REPO_URL --config-file $PYPIRC_PATH dist/* --skip-existing

FROM base as build
COPY . .
RUN poetry config experimental.new-installer false
RUN poetry install --no-dev
RUN pip3 --no-cache-dir install --upgrade awscli
RUN aws s3 cp s3://iotahoe-datascience/python_workers/dbpedia/output/ontologytagger-2021-05-23-20-49-40-099/output/model.tar.gz $ROOT/ontology_tagger
RUN mkdir $ROOT/ontology_tagger/bert-base-cased && cd $ROOT/ontology_tagger/bert-base-cased && \
wget https://huggingface.co/bert-base-cased/resolve/main/config.json && \
wget https://huggingface.co/bert-base-cased/resolve/main/tokenizer.json && \
wget https://huggingface.co/bert-base-cased/resolve/main/tokenizer_config.json 

FROM python:3.7-slim as production
ENV  ROOT=/home/worker/python/ontology_tagger \
     VIRTUAL_ENV=/home/worker/python/ontology_tagger/.venv\
     PATH=/home/worker/python/ontology_tagger/.venv/bin:/home/worker/python/ontology_tagger:$PATH
COPY --from=build /home/worker/python/ontology_tagger/pyproject.toml /home/worker/python/
COPY --from=build /home/worker/python/ontology_tagger/.venv /home/worker/python/ontology_tagger/.venv
COPY --from=build /home/worker/python/ontology_tagger/ontology_tagger /home/worker/python/ontology_tagger
WORKDIR $ROOT
ENV PYTHONPATH=$ROOT:/home/worker/python/
# args in format mongodb://user:pass@mongodb:27017/ops?authSource=admin,ops,jobs,pipelineId-ce6b9dad-0fa1-42f7-a04d-b54d1468dc7f:ontology_tagger_worker
ENTRYPOINT [ "primary_worker", "--mongo" ]

score 0 · Answer 1 · answered Aug 24 '21 at 12:50

0

The traceback

#16 1704.6   Looking in indexes: https://pkgs.dev.azure.com/iotahoe/Halo/_packaging/private-sources/pypi/simple
#16 1704.6   User for pkgs.dev.azure.com:
....
#16 1704.6     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/network/auth.py", line 256, in handle_401
#16 1704.6       username, password, save = self._prompt_for_password(parsed.netloc)
#16 1704.6     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/network/auth.py", line 226, in _prompt_for_password
#16 1704.6       username = ask_input("User for {}: ".format(netloc))
#16 1704.6     File "/home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 259, in ask_input
#16 1704.6       return input(message)

implies Pip is trying to ask for a password for pkgs.dev.azure.com - perhaps whatever <PAT> is isn't cutting it for Pip.

Looking at the line numbers in the traceback, this comes from a handle_401 function, so maybe double-check that those credentials indeed work for your package registry?

Failing that, you may need to set up an ~/.netrc file (maybe using Docker build secrets?) as explained in https://stackoverflow.com/a/55978589/51685 for Pip to read those credentials.

answered Aug 24 '21 at 12:50

AKX

152,115
15
115
172

I manually created a `~/.netrc` file. It's and the new Traceback error are appended to the post @AKX – Aug 25 '21 at 08:14
1

@StressedBoi_69420 Created a netrc file where? It's not going to help in your home directory; `docker build` is run in a container and won't see it. – AKX Aug 25 '21 at 08:15
In the same working directory as my `Dockerfile`. I'll update `ls` in post @AKX – Aug 25 '21 at 08:18
1

That doesn't help either unless it's available in the container (e.g. via `ADD`). However, it's _not secure_ to just ADD it into the dockerfile, since the credentials will then remain in the built image. Hence, Docker build secrets. See above. – AKX Aug 25 '21 at 08:19
Ah, will check out your "Docker build secrets" link now then. Ty – Aug 25 '21 at 08:20
So; I need a `mysecrets.txt` in the same `cwd`, each line being values for: ``, `` and ``? Which line(s) in `Dockerfile` do I append this to: `RUN --mount=type=secret,id=mysecret,dst=/ cat /`? – Aug 25 '21 at 08:41
1

You should be able to do something like `RUN --mount=type=secret,id=netrc,dst=~/.netrc poetry install` and run `docker build --secret id=netrc,src=./.netrc` ... – AKX Aug 25 '21 at 09:29
I placed that `RUN` line in `Dockerfile` (directly under `FROM python...`), and executed `docker build --secret id=netrc,src=./.netrc --no-cache -t my_app -f /home/danielbellio/.ssh/workers-python/workers/ontology_tagger/Dockerfile .` Error: `could not parse secrets: [id=netrc,src=./.netrc]: failed to stat ./.netrc: stat ./.netrc: no such file or directory` – Aug 25 '21 at 13:14

Dockerfile & Poetry | executor failed running [/bin/sh -c poetry install --no-dev]: exit code: 1

1 Answers1