Cannot install private dependency from artifact registry inside docker build

Question

I am trying to install a private python package that was uploaded to an artifact registry inside a docker container (to deploy it on cloudrun).

I have sucessfully used that package in a cloud function in the past, so I am sure the package works.

cloudbuild.yaml

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: [ 'build', '-t', 'gcr.io/${_PROJECT}/${_SERVICE_NAME}:$SHORT_SHA', '--network=cloudbuild', '.', '--progress=plain']

Dockerfile

FROM python:3.8.6-slim-buster

ENV APP_PATH=/usr/src/app
ENV PORT=8080

# Copy requirements.txt to the docker image and install packages
RUN apt-get update && apt-get install -y cython 

RUN pip install --upgrade pip

# Set the WORKDIR to be the folder
RUN mkdir -p $APP_PATH

COPY / $APP_PATH

WORKDIR $APP_PATH

RUN pip install -r requirements.txt --no-color
RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3 # This line is where the bug occurs


# Expose port 
EXPOSE $PORT

# Use gunicorn as the entrypoint
CMD exec gunicorn --bind 0.0.0.0:8080 app:app

The permissions I added are:

cloudbuild default service account (project-number@cloudbuild.gserviceaccount.com): Artifact Registry Reader
service account running the cloudbuild : Artifact Registry Reader
service account running the app: Artifact Registry Reader

The cloudbuild error:

Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
return func(self, options, args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 340, in run
requirement_set = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
if not criterion.candidates:
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__

you don't have more as error message? No clues about package missing, worng version, security access,... — guillaume blaquiere, May 15 '22 at 19:29
@guillaumeblaquiere No other errors than the ones I showed. Maybe adding the dependency with --extra-index-url like in cloud functions is not the right way to do this. — Benjamin Breton, May 15 '22 at 20:27
Did you try the same line directly in Cloud Build directly and not in docker build? — guillaume blaquiere, May 16 '22 at 08:24
I did not, but I am not sure how I could pip install directly from the cloudbuild into the docker python. — Benjamin Breton, May 16 '22 at 08:33
Do you experience no error pulling this private package from your terminal environment? — ErnestoC, May 16 '22 at 21:59
None, I use this package on multiple cloud functions, I just can’t find a way to pull it in cloud run — Benjamin Breton, May 17 '22 at 05:39

ErnestoC · Accepted Answer · 2022-05-19T17:13:59.367

From your traceback log, we can see that Cloud Build doesn't have the credentials to authenticate to the private repo:

Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception: //<-ASKING FOR USERNAME

I uploaded a simple package to a private Artifact Registry repo to test this out when building a container and also received the same message. Since you seem to be authenticating with a service account key, the username and password will need to be stored inside pip.conf:

pip.conf

[global]
extra-index-url = https://_json_key_base64:KEY@LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple/

This file therefore needs to be available during the build process. Multi-stage docker builds are very useful here to ensure the configuration keys are not exposed, since we can choose what files make it into the final image (configuration keys would only be present while used to download the packages from the private repo):

Sample Dockerfile

# Installing packages in a separate image
FROM python:3.8.6-slim-buster as pkg-build

# Target Python environment variable to bind to pip.conf
ENV PIP_CONFIG_FILE /pip.conf

WORKDIR /packages/
COPY requirements.txt /

# Copying the pip.conf key file only during package downloading
COPY ./config/pip.conf /pip.conf

# Packages are downloaded to the /packages/ directory
RUN pip download -r /requirements.txt
RUN pip download --extra-index-url https://LOCATION-python.pkg.dev/PROJECT/REPO/simple/ PACKAGES

# Final image that will be deployed
FROM python:3.8.6-slim-buster

ENV PYTHONUNBUFFERED True
ENV APP_HOME /app

WORKDIR /packages/
# Copying ONLY the packages from the previous build
COPY --from=pkg-build /packages/ /packages/

# Installing the packages from the copied files
RUN pip install --no-index --find-links=/packages/ /packages/*

WORKDIR $APP_HOME
COPY ./src/main.py ./

# Executing sample flask web app 
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app

I based the dockerfile above on this related thread, and I could confirm the packages were correctly downloaded from my private Artifact Registry repo, and also that the pip.conf file was not present in the resulting image.

But for that, I would need to commit the pip.conf containing the key to git. Would't that pose a security risk ? — Benjamin Breton, May 18 '22 at 05:50
In that case, a [multi-stage](https://docs.docker.com/develop/develop-images/multistage-build/#use-multi-stage-builds) dockerfile would be what you need to use to protect the `pip.conf` private key(s). I edited my answer to take this into account, let me know if it was helpful. — ErnestoC, May 18 '22 at 17:48
Did you run into any more problems with this solution? after checking the built container I could confirm no credentials made it into the deployed image in Cloud Run — ErnestoC, May 19 '22 at 17:28
I am still not sure I understand, all my code is hosted on a github repo, I do not want to host a private key on a repo, the multistage would allow me not to have the key on the container, but I would like to deploy on my cloudrun without a json key, I am sure there is a way to grant access from a service account. — Benjamin Breton, May 19 '22 at 19:08
Are you building your application from the github repo automatically with Cloud Build [triggers](https://cloud.google.com/build/docs/automating-builds/build-repos-from-github), or in a similar way? In your original question, you **did not** mention any of these very important details (where your code is hosted, how you run the builds, etc). This changes the way the [question should be answered](https://meta.stackexchange.com/questions/286803/), and you should go ahead and create a new question where you include all the relevant details about your CI/CD setup to get a relevant answer. — ErnestoC, May 19 '22 at 23:01
you are right, I have accepted your answer and created a separate issue here: https://stackoverflow.com/questions/72317003/cannot-install-private-dependency-from-artifact-registry-inside-docker-build-whe — Benjamin Breton, May 20 '22 at 10:21
@ErnestoC I just want to say, thanks! , after hours, you lead me to the solution!!! thanks! — nguaman, Aug 10 '22 at 11:35

score -1 · Answer 2 · answered May 25 '23 at 13:44

The best way of doing this is to mount a Docker secret on build.

To do it, you need to add this to your Dockerfile:

# Argument `GOOGLE_APPLICATION_CREDENTIALS` will be set from 
# Docker `/run/secrets/gsa_key` value
ARG GOOGLE_APPLICATION_CREDENTIALS=/run/secrets/gsa_key

# Install the keyring, needed to access Google Artifact Registry.
RUN pip install keyring keyrings.google-artifactregistry.auth

# Mount the secret (!) and install the private package.
RUN --mount=type=secret,id=gsa_key pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3

I.e., your Dockerifle at the end should look more or less like this:

FROM python:3.8.6-slim-buster

ENV APP_PATH=/usr/src/app
ENV PORT=8080

# Copy requirements.txt to the docker image and install packages
RUN apt-get update && apt-get install -y cython 

RUN pip install --upgrade pip

# Set the WORKDIR to be the folder
RUN mkdir -p $APP_PATH

COPY / $APP_PATH

WORKDIR $APP_PATH

# Argument `GOOGLE_APPLICATION_CREDENTIALS` will be set from 
# Docker `/run/secrets/gsa_key` value
ARG GOOGLE_APPLICATION_CREDENTIALS=/run/secrets/gsa_key

RUN pip install -r requirements.txt --no-color

# Install the keyring, needed to access Google Artifact Registry.
RUN pip install keyring keyrings.google-artifactregistry.auth

# Mount the secret (!) and install the private package.
RUN --mount=type=secret,id=gsa_key pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3

# Expose port 
EXPOSE $PORT

# Use gunicorn as the entrypoint
CMD exec gunicorn --bind 0.0.0.0:8080 app:app

And with that in place, you will build your Docker image with this command:

DOCKER_BUILDKIT=1 docker build \
  --secret id=gsa_key,src=/path/to/your/google/credentials.json \
  -t my-application .

Just adapt that to your cloudbuild.yaml and you should be fine. Also, notice that you don't need to do multistage as the secret is never shown in the Docker image.

P.D.: You will need a modern Docker version, and Docker Buildkit enabled.

Cannot install private dependency from artifact registry inside docker build

2 Answers2

Linked