17

When building a container (through docker build, docker run or docker-compose) using NVIDIA containers, I get the following error, somewhat randomly. When I usually start building the container, it works the first time. But next few times usually don't work.

------
 > [internal] load metadata for nvcr.io/nvidia/l4t-base:r32.5.0:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: 
  failed to create LLB definition: failed to authorize: 
  rpc error: code = Unknown desc = failed to fetch anonymous token: unexpected status: 401 Unauthorized

My current workaround is to use a slightly different image temporarily whilst I’m adjusting other parts of the Dockerfile.

Ben Butterworth
  • 22,056
  • 10
  • 114
  • 167

4 Answers4

28

You have to docker login nvcr.io

Most of this is documented in the setup docs, unfortunately nvcr.io doesn't have very good SEO or UX, so search results and most attempts don't show this page. This is, confusingly, different to ngc catalog / docker login ngc.nvidia.com.

  • Install NGC CLI: (for macOS)
    • Run curl -O https://ngc.nvidia.com/downloads/ngccli_mac.zip && unzip ngccli_mac.zip && chmod u+x ngc
    • mv ngc /usr/local/bin/ngc
  • Setup your environment:
    • Create an account and get your API key from https://ngc.nvidia.com/
    • Add the API key to the CLI: ngc config set
    • Login to registry: docker login nvcr.io
  • Observe docker CLI output containing correct auth details:
[auth] nvidia/tensorrt:pull,push token for nvcr.io

Tips

Make sure you login to the right container registry. It's in the name of the docker image you are using. e.g. nvcr.io/nvidia/tensorrt:22.01-py3 uses nvcr.io.

Explanation

With the benefit of hindsight, I understood failed to fetch anonymous token: unexpected status: 401 Unauthorized. I was not authenticated (or more correctly, I was authenticated to Docker Hub and NGC Catalog instead of nvcr.io). It was giving me an anonymous token, which hit the rate limit. I needed to create an account to get higher limits.

Ben Butterworth
  • 22,056
  • 10
  • 114
  • 167
  • Thanks. I solved issue with this answer. I had a little difficulty to login to registry. Following contents helped me to login. https://docs.nvidia.com/dgx/ngc-registry-for-dgx-user-guide/index.html https://catalog.ngc.nvidia.com/ – jaylee Aug 08 '22 at 15:30
  • 1
    To get your API key, after login, click on the right top corner -> setup. get your API key – Shiwakant Bharti Dec 25 '22 at 20:13
  • I had to chase up this as well. https://stackoverflow.com/questions/71770693/error-saving-credentials-error-storing-credentials-err-exit-status-1-out – Shiwakant Bharti Dec 25 '22 at 21:39
8

Restarting docker server solved the issue

sudo systemctl restart docker
vozman
  • 1,198
  • 1
  • 14
  • 19
  • Thanks! It helped in my case. Was getting this error while building a tritonserver image. ### ERROR: failed to solve: failed to fetch anonymous token: unexpected status: 401 Unauthorized error: build failed ### – Girish Sharma Jul 08 '23 at 11:36
1

If you happen to get this error and it has worked before, you probably have run into a download limit from nvcr. To circumvent it, it suffices to just restart docker.

For Docker desktop click on the docker icon in the taskbar and click restart. Or on Mac your can use cmd + r.

Docker Restart

Error404
  • 84
  • 7
  • 1
    I don't think this would work, because the rate limit is server side, restarting software on your machine doesn't do anything. I think you've tricked yourself when debugging, the rate limit timeout finished. – Ben Butterworth Feb 22 '23 at 12:32
  • Then maybe it was not the rate limit. But i waited for over an hour before I knew this trick and now it takes two minutes. I'll leave it here maybe it helps someone else. – Error404 Feb 22 '23 at 13:08
  • 1
    Agree bouncing Docker worked for me too, NO idea why. – JeffHeaton Mar 25 '23 at 02:48
0

Updating Docker from my version (4.5.0) to the latest version (4.9.0) helped me.

Asya Lyanova
  • 49
  • 2