How to solve permissions for push to Google Artifact Registry from Cloud Build using jib-maven-plugin?

Question

This problem seems to be all about my permissions in GCP, but attempting to set the right permissions so far has not worked.

I'm using com.google.cloud.tools:jib-maven-plugin to package a Spring Boot project into a container and push it to Google Artifact Registry (GAR). It works just fine when I run it locally, but it fails when I run the maven build with Google Cloud Build. It says it fails because artifactregistry.repositories.downloadArtifacts permission is missing.

But this is one of the permissions enabled by default according to Google Docs.

My target is a Google Artifacts (docker) Registry. I'm able to change the target to a Google Container Registry (deprecated, so I need to change to GAR) and that works fine under Cloud Build, no permission problems there. The build also downloads jar files from a maven repository stored in a different GAR, though in the same Google project. So clearly it is okay with permissions for that maven GAR.

I verified that running the maven build locally works, including writing to the GAR, which eliminates something going bad in the jib plugin configuration, or in the GAR configuration. This is using my own user credentials.

What have I tried?

• I added the appropriate roles to the default service account Cloud Build uses (though they are supposed to be there anyway). The downloadArtifacts permission is included in role Artifact Registry Reader, so I added that role as well as Artifact Registry Writer.
• I switched to a different Service Account (build service account, let's call it BSA) and, yes, made sure that had the same appropriate roles (see above).
• I added the BSA as a principal to the target GAR and gave it the appropriate roles there as well (getting desperate)
• My user credentials include Owner role so I added Owner role to the BSA (not something I want to keep)

All of these gave me the same permission denied error. Just in case I had really misunderstood something I added a step to my Cloud Build yaml to runs gcloud info and verified that, yes, it is using the BSA I have configured with the roles I need.

Is there something I missed? Thanks

...Edit More info. Most of my builds use jib but one uses Spotify to create a local docker image and then uses docker to push to the registry. And this works! So the problem is specific to jib. Somehow, under cloud build, jib is not seeing the creds, though it does see them locally.

...Edit The actual error message:

Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.6.1:build (build-and-push-docker-image) on project knifethrower: Build image failed, perhaps you should make sure you have permissions for australia-southeast1-docker.pkg.dev/redacted/bonanza-platform/knifethrower and set correct credentials. See https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#what-should-i-do-when-the-registry-responds-with-forbidden-or-denied for help: Unauthorized for australia-southeast1-docker.pkg.dev/redacted/bonanza-platform/knifethrower: 403 Forbidden
[ERROR] {"errors":[{"code":"DENIED","message":"Permission \"artifactregistry.repositories.downloadArtifacts\" denied on resource \"projects/redacted/locations/australia-southeast1/repositories/bonanza-platform\" (or it may not exist)"}]}
[ERROR] -> [Help 1]

Also note I'm using the latest version of jib: 3.1.4

...Edit

I've tried a couple more things. I added an earlier step, before the maven build, which does a gcloud auth configure-docker --quiet --verbosity=debug australia-southeast1-docker.pkg.dev. That creates a /builder/home/.docker/config.json file. Because there seems to be confusion about where that file should really live I copied it to /root/.docker. But that did not help.

The second thing I tried was using the $DOCKER_CONFIG to point to the /builder/home/.docker directory (as suggested here) and that did not help either.

Same error in both cases. I do get a message from the gcloud auth configure-docker...

WARNING: `docker` not in system PATH.
`docker` and `docker-credential-gcloud` need to be in the same PATH in order to work correctly together.
gcloud's Docker credential helper can be configured but it will not work until this is corrected.
Adding credentials for: australia-southeast1-docker.pkg.dev
Docker configuration file updated.
INFO: Display format: "default"

I think this is trying to be helpful and tell me it cannot find docker installed (which is true) but it created the creds anyway (also true). And it should not matter because jib doesn't rely on docker itself, just the creds. However it still doesn't work.

Answer 1

Part of the problem (and thanks again to @ChanseokOh for flagging this) is that I was still using jib 1.6.1 when I thought I was using 3.1.4. Changing that plus some other stuff fixed the problem. So here's the full story for the next person who struggles with this:

First, this is what my pom file has:

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <version>3.1.4</version>
    <configuration>
        <from>
            <image>${base.image}</image>
        </from>
        <to>
            <image>${docker.image.repo}/${project.artifactId}:latest</image>
            <tags>
                <tag>${VERSION_ID}</tag>
                <tag>latest</tag>
            </tags>
        </to>
        <creationTime>USE_CURRENT_TIMESTAMP</creationTime>
        <allowInsecureRegistries>true</allowInsecureRegistries>
        <container>
            <ports>
                <port>8080</port>
            </ports>
        </container>
    </configuration>
    <executions>
        <execution>
            <id>build-and-push-docker-image</id>
            <phase>package</phase>
            <goals>
                <goal>build</goal>
            </goals>
        </execution>
    </executions>
</plugin>

The version is important. Although the old version (1.6.1) worked just fine locally it did not work on Cloud Build.

My Cloud Build file looks like this:

...
  - name: 'gcr.io/cloud-builders/gcloud'
    args:
      - '-c'
      - >
        gcloud auth configure-docker --quiet --verbosity=debug `echo
        ${_CONTAINER_REPO} | cut -d / -f 1` 
        /root 
    id: gcloud auth
    entrypoint: /bin/bash
...
  - name: 'gcr.io/cloud-builders/mvn:3.5.0-jdk-8'
    args:
      - '-Dmaven.test.skip=false'
      - '-Dmaven.repo.local=/workspace/.m2/repository'
      - '--settings'
      - custom-settings.xml
      - clean
      - install
      - '-DskipITs'
      - '-B'
      - '-X'
      - '-DVERSION_ID=$TAG_NAME'
      - '-DBRANCH_ID=master'
      - '-DPROJECT_ID=$PROJECT_ID'
      - '-DCONTAINER_REPO=${_CONTAINER_REPO}'
      - '-DMAVEN_REPO=${_MAVEN_REPO}'
      - '-DDOCKER_CONFIG=/builder/home/.docker'
      - '-P'
      - release
    id: build

The gcloud auth step gets the docker credentials file created.

The next step is the maven build and for that the trick is to define DOCKER_CONFIG pointing to the correct location of the docker creds file. I believe creating the docker creds file, defining DOCKER_CONFIG and getting the version number right are all required for the solution.

An interesting aside is that gcr.io/cloud-builders/gcloud and gcr.io/cloud-builders/mvn:3.5.0-jdk-8 still reference Google Container Repository which is superseded by Artifact Repository, and is the whole reason I got into this but I have not seen any updated reference for these images. Docs are here.

Answer 2

Adding another answer for the sake of completeness:

In the basic case, Jib may work out of the box on Google Cloud Build (GCB) without manually configuring credentials for Jib when pushing to your Google Container Registry (GCR) (mostly when pushing to GCR in the same GCP project). This is because Jib may automatically pick up the Application Default Credentials (ADC) that come from the environment where GCB runs. However, I am not sure if the ADC on GCB also works for Artifact Registry out of the box.

Of course, you can always configure registry credentials for Jib in many different ways.