11

I have two docker repositories running on the same JFrog cloud account/instance. One for internal release candidates and the other for potentially external GC releases. I want to be able to build the docker images and push to the internal repository, let QA/UAT go to town, and then copy the image to the release repository. I don't want to rebuild the image from source. Unfortunately, when I try to pull, tag and then push the image, I'm getting an error:

unauthorized: Pushing Docker images with manifest v2 schema 1 to this repository is blocked.

Both repositories block schema 1 manifests, but I am pushing fine to the internal repository, so it doesn't make much sense I wouldn't be able to push the same image to the release repository.

I've setup a pretty simple test to confirm (actual repository URLs censored):

% docker pull hello-world:latest
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
...
% docker tag hello-world:latest internal-rc.jfrog.io/hello-world:1.0.0-beta
% docker push internal-rc.jfrog.io/hello-world:1.0.0-beta
The push refers to repository [internal-rc.jfrog.io/hello-world]
9c27e219663c: Pushed
...
% docker system prune -a
...
Total reclaimed space: 131.8MB
% docker image pull internal-rc.jfrog.io/hello-world:1.0.0-beta
1.0.0-beta: Pulling from hello-world
0e03bdcc26d7: Pull complete
...
% docker image tag internal-rc.jfrog.io/hello-world:1.0.0-beta docker-release.jfrog.io/hello-world:1.0.0
% docker image push docker-release.jfrog.io/hello-world:1.0.0
The push refers to repository [docker-release.jfrog.io/hello-world]
9c27e219663c: Layer already exists
[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the docker-release.jfrog.io registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
unauthorized: Pushing Docker images with manifest v2 schema 1 to this repository is blocked. For more information visit https://www.jfrog.com/confluence/display/RTF/Advanced+Topics#AdvancedTopics-DockerManifestV2Schema1Deprecation

So I can upload the image fine to the first repository, and confirm that it is using schema 2:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 7004,
    "digest": "sha256:66f750f4871ba45724699d7341ee7135caba46f63fb205351197464a66b55eff"
...

Does that mediaType being v1 matter? It seems like the manifest itself is version 2... But I don't know how I would change that, or why it would be allowed in one repository but not the other.

I'm using I believe the latest version of docker Docker version 19.03.8, build afacb8b

Anyone have any idea what's going on there? Is the schema version being changed between when I upload it the first time and when I download it? Or is it when I tag it or upload it the second time?

DrTeeth
  • 927
  • 2
  • 10
  • 32

4 Answers4

19

The root of this problem can probably be classified as user error. Specifically the user I'm using somehow had permissions removed from the release repository. Once that was restored everything works as expected.

I say "probably" because the error message has nothing to do with the actual problem, and cost me 2-3 hours worth of wild goose chasing.

So... If you see this error, go ahead and double check everything else around permissions/access before trying to figure out if there's something actually wrong with your image schema version.

DrTeeth
  • 927
  • 2
  • 10
  • 32
  • 4
    Oof. I just ran into this and turns out write permissions were missing. Thanks for saving me those 2-3 hours of wild goose chasing. Appreciated! – moertel Jul 17 '20 at 07:57
  • 2
    Thanks. That saved me some time, too. Afterwards I also found the [JFrog issue](https://www.jfrog.com/jira/browse/RTFACT-23722) for that. – Christian Feb 04 '21 at 08:00
  • 1
    I gave all the permission to user for repository still I have the same problem. ```Error response from daemon: unauthorized: Pulling Docker images with manifest v2 schema 1 to this repository is blocked.``` From CentOS7 in docker version 1.13.1 I'm able to pull the image but same image in Ubuntu on docker version 18.09.7 doesn't work. – Mahboob May 07 '21 at 13:13
  • Unfortunately, I no longer have access to JFrog (switched jobs, no longer dealing with it). But that's quite a few differences between those environments (CentOS/1.13 vs Ubuntu 18.09). Offhand I'd try to really make sure that both are authenticating exactly the same way with exactly the same user. I'd also put in a plug for Cloudsmith - we wound up switching to them because they were cheaper, easier and their support team was awesome. – DrTeeth May 07 '21 at 19:25
8

We had a different case today with a similar error. I'm adding here because this is the top google result at the moment.

Pulling Docker images with manifest v2 schema 1 to this repository is blocked.

The fix was to change a setting on the remote repository.

Via UI: Artifactory Admin -> Repositories -> Repositories -> Remote tab

Then select your Docker Hub repo, whatever you named it, then under Basic settings -> Docker Settings, uncheck the checkbox labeled

Block pulling of image manifest v2 schema 1

After that our images began pulling properly again.

There is a similar checkbox on local repos for pushing.

For what it's worth, we're on Artifactory version 7.18.5 rev 71805900

edit: The surprisingness of our particular issue is (potentially) explained in some more detail here: https://www.jfrog.com/jira/browse/RTFACT-2591

Docker pull requests fail due to a change in Docker Hub behavior. Now Docker Hub HTTP response headers return in lower case, for example, 'content-type' instead of 'Content-Type', causing Artifactory to fail to download and cache Docker images from Docker Hub.

but we have not yet tested whether an upgrade allows us to re-enable the aforementioned checkbox.

DeeBee
  • 299
  • 3
  • 4
0

I have been getting the below errors while either pulling or pushing docker images from build servers. i have a proxy in env which used to connect docker registry. My DNS server while resolving the proxy FQDN it was returning a non functional IP-address. i have 4 DNS servers and multiple proxy servers based on region. Once DNS is updated and working/functional proxy returned it started working. Just check on network side too , it may solve the issue. the error messages were misleading initially i though to be docker layer issue, credential issue ..but no network issue. for the below errors

error pulling image configuration: unknown blob or

[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the docker registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/ manifest invalid: manifest invalid . Will start No.6 try.

TheFixer
  • 89
  • 1
  • 2
0

I got this error as well. Turned out that my pipeline was trying to push an image to jFrog with a name that already existed in the jFrog repo. I added CI_COMMIT_SHORT_SHA as a suffix to the tag which solved the problem for me.

The error in my case was misleading. I did verify the manifest versions of all images involved. That is when it dawned on me that this error description may not be accurate.

I hope this saves someone some time.

Swifty
  • 1,422
  • 2
  • 18
  • 38