2

I have an app in Angular/Nest.js deployed on Azure. It has been working fine for the last few months but today it suddenly started showing these errors:

On frontend it is showing CORS errors but on the backend I see this:

2022-06-06T06:06:39.208Z INFO  - Starting multi-container app..
2022-06-06T06:06:39.556Z INFO  - Pulling image: example.azurecr.io/example_backend_uat_github:latest
2022-06-06T06:06:42.254Z INFO  - latest Pulling from example_backend_uat_github
2022-06-06T06:06:42.255Z INFO  -  Digest: sha256:fbc2we4g67c865c345567098775ed5fec4aecd7285e4fce947223d3262c901de
2022-06-06T06:06:42.256Z INFO  -  Status: Image is up to date for example.azurecr.io/example_backend_uat_github:latest
2022-06-06T06:06:42.258Z INFO  - Pull Image successful, Time taken: 0 Minutes and 2 Seconds
2022-06-06T06:06:42.270Z INFO  - Starting container for site
2022-06-06T06:06:42.271Z INFO  - docker run -d -p 1496:3000 --name example-portal-backend-uat_backend_0_01449c14 -e WEBSITE_CORS_ALLOWED_ORIGINS=https://example-portal-uat.azurewebsites.net -e WEBSITE_CORS_SUPPORT_CREDENTIALS=True -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITES_PORT=3000 -e WEBSITE_SITE_NAME=example-portal-backend-uat -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=example-portal-backend-uat.azurewebsites.net -e WEBSITE_INSTANCE_ID=some-random-id -e HTTP_LOGGING_ENABLED=1 -e WEBSITE_USE_DIAGNOSTIC_SERVER=False example.azurecr.io/example_backend_uat_github:latest
2022-06-06T06:06:42.334Z INFO  - Pulling image: redis
2022-06-06T06:06:44.051Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://registry-1.docker.io/v2/library/redis/manifests/latest: unauthorized: incorrect username or password"}
2022-06-06T06:06:44.052Z ERROR - Pulling docker image redis failed:
2022-06-06T06:06:44.053Z WARN  - Image pull failed. Defaulting to local copy if present.
2022-06-06T06:06:44.056Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2022-06-06T06:06:44.057Z ERROR - multi-container unit was not started successfully
2022-06-06T06:06:44.059Z INFO  - Container logs from example-portal-backend-uat_backend_0_01449c14 =
2022-06-06T06:06:44.088Z INFO  - Container logs from example-portal-backend-uat_redis_0_01449c14 =
2022-06-06T06:06:44.337Z INFO  - Stopping site example-portal-backend-uat because it failed during startup.

We didn't make any update. I saw a similar post and I verified that:

  • In my Container Register, I have Admin enabled
  • I have set DOCKER_REGISTRY_SERVER_PASSWORD, DOCKER_REGISTRY_SERVER_URL and DOCKER_REGISTRY_SERVER_USERNAME in my App Service Configuration

Is there anything else I should check?

Tom Bom
  • 1,589
  • 4
  • 15
  • 38
  • Have you enabled App Service Identity and granted access to ACR? https://stackoverflow.com/questions/60163440/docker-fails-to-pull-the-image-from-within-azure-app-service – Andriy Bilous Jun 06 '22 at 10:39
  • Does this answer your question? [Docker fails to pull the image from within Azure App Service](https://stackoverflow.com/questions/60163440/docker-fails-to-pull-the-image-from-within-azure-app-service) – Andriy Bilous Jun 06 '22 at 10:40
  • @AndriyBilous I don't understand how to do step 4. I enabled App Service Identity but I don't see how to grand access to ACR. What troubles me is that before this weekend it was working fine. – Tom Bom Jun 06 '22 at 10:56
  • 1
    Hi Tom. We have had similar issues this morning with our multi-container app services. We use the datadog agent. If we pull from https://hub.docker.com/r/datadog/agent we get the same error. If we pull from gcr.io/datadoghq/agent we get a DockerApiException internal server error - Unauthorised Exception. Both of these images are public and have been working for many months, until some point this morning. We haven't found a solution yet apart from to deploy without the agent as a temporary solution. Is this a wider Azure issue? – SamJolly Jun 06 '22 at 11:21
  • Go to the Azure container registry and add role assignment to app service identity(it should have name of your App Service) with permissions of AcrPull. – Andriy Bilous Jun 06 '22 at 11:25
  • @SamJolly I don't know and it's driving me crazy because I have absolutely no idea how to proceed and what to do about it. I see people with similar issues on their initial deployment but this was a fully functional app. I don't even know how to deploy without an agent and what to do – Tom Bom Jun 06 '22 at 16:23
  • @SamJolly I was with Azure support and they fixed it. I had to update my docker-compose file, for redis: image:. It used to be "redis" but I had to change it to the repository link – Tom Bom Jun 07 '22 at 06:41
  • @TomBom Can you give me an example of the new repository link for redis? Glad Azure Support could help you – SamJolly Jun 07 '22 at 08:11
  • Hi @TomBom I have the same problem, can you share your image link ? – waltexwq Jun 08 '22 at 11:02
  • Same issue here with redis:latest. All was fine until this morning. If there's a change to do to the docker-compose configuration file, I'd be glad to have a link about that too. – WhyNotTryCalmer Jun 09 '22 at 08:51
  • We placed the image in a private repository on Azure, and then set the redit url to that. I don't know the exact process how we did it (lots of try and fail), but the last bit was helped by the Azure support team. – Tom Bom Jun 09 '22 at 13:07
  • @TomBom Ah you mean that instead of pulling the public redis image from your application service, you stored that image to your container registry, so that docker-compose would fetch it from there when building instead of pulling it from the web? – WhyNotTryCalmer Jun 09 '22 at 13:10
  • I can confirm that changing the target image (redis:latest) into my private registry and referencing it from the docker-compose (from `redis: image: redis:latest` to `redis: image: `) fixed it for me. Apparently pulling images from an external provider no longer works. I am not sure if that's just a bug on Azure's end or a known limitation coming into force. In any case that's a serious issue for CI/CD. – WhyNotTryCalmer Jun 09 '22 at 13:42

1 Answers1

0

Apparently Microsoft Azure changed the way multi-container apps authenticate to third-party images providers. Pending a more long-term solution, you can change references to images to use an internal reference. For example for redis that would be:

docker-compose.yaml
-------------------
  services:
    redis: <path/to/your/ACR/image:version>
    # instead of pulling from a public repo:
    # redis: redis:latest

Of course you need to push the third-party images to a private registry first.

WhyNotTryCalmer
  • 357
  • 5
  • 15