Docker Trust: could not rotate trust to a new trusted root: failed to validate data with current trusted certificates

Question

I've suddenly got this message after a month of docker trust working fine for me via GitLab CI.

I have a Gitlab Runner that mounts the ~/.docker/trust (so its persisted) and pushes it to our QA registry.

tag_image_test:
  stage: tag_image
  script:
    - docker login -u "gitlab-ci-token" -p "$CI_BUILD_TOKEN" $CI_REGISTRY
    - docker pull "${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}"
    - export DOCKER_CONTENT_TRUST=1
    - export DOCKER_CONTENT_TRUST_SERVER=$QA_REGISTRY_SIGNER
    - export DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE=$QA_REGISTRY_SIGNER_ROOT_PASSPHRASE
    - export DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE=$QA_REGISTRY_SIGNER_REPO_PASSPHRASE
    - docker login -u "$QA_REGISTRY_USERNAME" -p "$QA_REGISTRY_PASSWORD" $QA_REGISTRY_URL
    - export PROJ_PATH=$(echo -en $CI_PROJECT_PATH | tr '[:upper:]' '[:lower:]')
    - docker tag "${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}" "${QA_REGISTRY_IMAGE}/${PROJ_PATH}:${CI_COMMIT_REF_SLUG}"
    - docker push "${QA_REGISTRY_IMAGE}/${PROJ_PATH}:${CI_COMMIT_REF_SLUG}"

However the push commands ends with:

time="2019-03-18T11:51:14Z" level=debug msg="failed to verify TUF data for: qa.registry.local/mygroup/myimage, valid signatures did not meet threshold for "
time="2019-03-18T11:51:14Z" level=debug msg="downloaded 1.root is invalid: could not rotate trust to a new trusted root: failed to validate data with current trusted certificates"
time="2019-03-18T11:51:14Z" level=debug msg="Client Update (Root): could not rotate trust to a new trusted root: failed to validate data with current trusted certificates"
could not rotate trust to a new trusted root: failed to validate data with current trusted certificates

When I look at the root.json file, the expiry is not for a long time:

"expires":"2029-02-08T15:07:05.172338131Z"

Same for targets.json:

"expires":"2022-02-10T15:07:05.173954376Z"

So I'm at a loss for what is going on and probably don't understand what it is trying to do. Does anyone have any insight?

Is this issue https://github.com/theupdateframework/notary/issues/1281 ? — howie, Mar 26 '19 at 13:12

score 2 · Answer 1 · 2019-03-23T17:56:57.123

I’m still learning docker, but are you sure it is root.json that it is looking in and not roots.json.

Based on the configuration here, it should be looking in roots.json for the trusted certs.

Maybe you are pushing to the wrong file to identify your roots, or you could just have a typo in your post.

In any case, this is helpful: https://github.com/cirocosta/docker-cli/blob/master/vendor/github.com/theupdateframework/notary/trustpinning/certs.go

How those errors are generated are seen there with comments for why those errors occur.

For example, regarding your key rotation error:

// ErrRootRotationFail is returned when we fail to do a full root key rotation // by either failing to add the new root certificate, or delete the old ones

Thanks for pointing out the source, gives me some hints. The root.json was just a typo on my part. — Tim, Mar 28 '19 at 11:26

Faiz Fareed · Answer 2 · 2019-03-23T18:33:24.970

it's only a locally corrupted state right? You should be able to fix it with a notary remove server.example.com/test1.

The fix I want to get in for this is lazy initialization where one no longer has to explicitly call notary init. As part of lazy initialization, we would always query the server for existing data before assuming it needs to be created locally.

A shorter term fix may be to check the server, or if network connectivity isn't available, the local cache, for existing data. At the moment I believe init assumes the repo doesn't exist and overwrites any existing cache.

Also please make sure to configure DNS and made host entry in host file.

For the purposes of UCP Signing Policy, configured via the “Content Trust” section of the Admin Settings, it’s necessary that we can identify the image was signed by a member of the UCP organization. We do that by making use of client bundles that you can download for your user account from UCP. Client Bundles contain a “cert.pem” file which is an x509 certificate signed by the UCP Certificate Authority, and a “key.pem” file which is the private key matched with the certificate.

You need to create the “targets/releases” delegation and one other delegation, e.g. “targets/my_user” and add the “cert.pem” as the public signing key to both. When another service then inspects the trust data, they can determine that the certificate belongs to a member of the UCP organization and their signatures should be trusted. You then need to import the key.pem so it is available for signing when you push.

The documentation 23 provides more information and specific commands to run, specifically the “Initialize a Repo” section.

Docker Trust: could not rotate trust to a new trusted root: failed to validate data with current trusted certificates

2 Answers2