0

I was following this StackOverflow answer (https://stackoverflow.com/a/58021389/11268971) to get my Docker build image process to use a cache directory with package wheels for pip install. However, when I switched from (suggested in the SO answer)

# syntax = docker/dockerfile:experimental
FROM python:3.7
RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt

to

# syntax = docker/dockerfile:experimental
FROM python:3.7
RUN --mount=type=cache,target=/home/my_username/new_dir_i_created_just_for_docker_builds pip install --no-index --find-links=/home/my_username/new_dir_i_created_just_for_docker_builds -r requirements.txt

Docker buildkit started failing to mount the new directory, it still looks for packages in the old directory I mounted /root/.cache/pip.

I tried clearing the cache with docker builder prune and its different flags. To no avail.

Any ideas how I can get this to work?

P. S. I decided to create my own directory under my user, since installing /root/.cache/pip would regularly stop (docker build would download packages from the Net, instead of installing cached versions) after the first 4 or so builds or 1 day after initial cache mount. My hypothesis is that /root/.cache/pip was overwritten or invalidated.

EDIT: My Dockerfile

# syntax = docker/dockerfile:experimental
FROM python:3.7
ADD . /app
WORKDIR /app
RUN --mount=type=cache,target=/home/my_username/new_dir_i_created_just_for_docker_builds pip install --no-index --find-links=/home/my_username/new_dir_i_created_just_for_docker_builds -r requirements.txt
ENV CONTAINER_ID=""
ENV HOST=""
ENV PORT=""
ENV MODELPATH=""
CMD "python" "/app/model.py" $CONTAINER_ID $HOST $PORT $MODELPATH

My Docker build logs:

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 468B done
#2 DONE 0.1s

#3 resolve image config for docker.io/docker/dockerfile:experimental
#3 DONE 3.0s

#4 docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2f3a...
#4 CACHED

#5 [internal] load metadata for docker.io/library/python:3.7
#5 DONE 0.0s

#7 [internal] load build context
#7 transferring context: 1.00kB done
#7 DONE 0.0s

#6 [stage-0 1/4] FROM docker.io/library/python:3.7
#6 CACHED

#8 [stage-0 2/4] ADD . /app
#8 DONE 1.2s

#9 [stage-0 3/4] WORKDIR /app
#9 DONE 0.2s

#10 [stage-0 4/4] RUN --mount=type=cache,target=/home/my_username/new_dir_i_created_just_for_docker_builds ...
#10 3.110 Looking in links: /home/my_username/new_dir_i_created_just_for_docker_builds/
#10 3.119 ERROR: Could not find a version that satisfies the requirement absl-py==0.9.0 (from -r /app/requirements.txt (line 1)) (from versions: none)
#10 3.119 ERROR: No matching distribution found for absl-py==0.9.0 (from -r /app/requirements.txt (line 1))
#10 ERROR: executor failed running [/bin/sh -c pip install --no-index --find-links=/home/my_username/new_dir_i_created_just_for_docker_builds -r /app/requirements.txt]: runc did not terminate sucessfully
------
 > [stage-0 4/4] RUN --mount=type=cache,target=/home/my_username/new_dir_i_created_just_for_docker_builds pip install --no-index --find-links=/home/rytis/test-cont-cache/ -r /app/requirements.txt:
------
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: executor failed running [/bin/sh -c pip install --no-index --find-links=/home/my_username/new_dir_i_created_just_for_docker_builds/ -r /app/requirements.txt]: runc did not terminate sucessfully
rkj
  • 329
  • 1
  • 3
  • 11
  • **1.** I never see the cache expired issue... If possible any special setting make docker clean this for you? E.g. low disk storage? **2.** I failed to understand your logic: you specify `--no-index` means you won't search pypi, but to find package from the `--find-links`, but there is nothing in your `/home/my_username/new_dir_i_created_just_for_docker_builds` as it's just a cache directory. In fact I can't understand how you get above test results. If possible you miss some steps in your question? Could you paste full commands and its logs one by one? – atline Jul 23 '20 at 08:46
  • I have added all my wheels and tar.gz to `/home/my_username/new_dir_i_created_just_for_docker_builds`. So it has for example: six-1.15.0-py2.py3-none-any.whl, absl-py-0.9.0.tar.gz – rkj Jul 23 '20 at 08:49
  • How you manage that? `target=/home/my_username/new_dir_i_created_just_for_docker_builds` not means the host folder, but the folder in build container, while its corresponding host folder is maintain by docker daemon in other location, it's not the path of `home/my_username/new_dir_i_created_just_for_docker_builds` on the host, so how you could put wheel in that folder in advance? – atline Jul 23 '20 at 08:51
  • my goal is to prepopulate the cache first in some host directory and then have Docker build install from it in the image, so that even the first build is fast. – rkj Jul 23 '20 at 08:54
  • Yes, I understand your purpose. First, I suppose you know `docker build` will start a `build container`, then user can just specify the path of the cache in build container, while the populated path in host was not controlled by user. You have no chance to preinstall things in that cache directory. – atline Jul 23 '20 at 08:59
  • I added my docker build logs, too – rkj Jul 23 '20 at 09:03
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/218420/discussion-between-rytis-jonynas-and-atline). – rkj Jul 23 '20 at 09:03

0 Answers0