29

I am trying to upgrade one of our docker base images to the latest stable version of Ubuntu. I have isolated the problem to a simple reproducible case. I have a Dockerfile like this:

FROM ubuntu:22.04

MAINTAINER mep-dev@zulily.com

# Install java and clean-up
RUN apt-get update

When I build it on my local machine, I don't have any problems. However, when I build it on my CICD, I sometimes get this error:

Step 3/3 : RUN apt-get update
---> Running in 6ca01b60de64
Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [109 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [99.8 kB]
Err:1 http://archive.ubuntu.com/ubuntu jammy InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Err:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Err:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Err:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Reading package lists...
W: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy InRelease' is not signed.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy-updates InRelease' is not signed.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy-backports InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy-backports InRelease' is not signed.
W: http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://security.ubuntu.com/ubuntu jammy-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://security.ubuntu.com/ubuntu jammy-security InRelease' is not signed.
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/.deb /var/cache/apt/archives/partial/.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
The command '/bin/sh -c apt-get update' returned a non-zero code: 100

If I comment out RUN apt-get update, then it succeeds, and I can enter the container and see that /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg and /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg do exist and have read for all permissions:

root@b778220b39d8:/# ls -l /etc/apt/trusted.gpg.d
total 8
-rw-r--r-- 1 root root 2794 Mar 26 2021 ubuntu-keyring-2012-cdimage.gpg
-rw-r--r-- 1 root root 1733 Mar 26 2021 ubuntu-keyring-2018-archive.gpg

I also checked the parent directories, and they have r-x at least for all.

This answer might be related, but why would the file have the correct structure when the base image is used in one environment and not another?

Update:

By using --pull, I can see the exact image it's using.

$ docker build --pull -t $EMAIL_DELIVERABILITY_ARN .
Step 1/3 : FROM ubuntu:22.04
22.04: Pulling from library/ubuntu
Digest: sha256:b6b83d3c331794420340093eb706a6f152d9c1fa51b262d9bf34594887c2c7ac
Status: Downloaded newer image for ubuntu:22.04
---> 27941809078c

This is the same sha and image id that I see when building locally, which works.

I am not having the same problem with ubuntu:20.04.

morningstar
  • 8,952
  • 6
  • 31
  • 42
  • 2
    Is there any chance your CICD environment has a stale version of the image? Can you explicitly `docker pull ubuntu:22.04` before trying to build your image? – larsks Jun 15 '22 at 00:59
  • @larsks I think you have the right answer. Want to make it an answer so you can get the reputation points for it? – morningstar Jun 15 '22 at 16:46
  • Can you tell us anything about your CI/CD environment? – larsks Jun 15 '22 at 22:50
  • which version of Docker do you have ? – Tuxity Jun 17 '22 at 10:17
  • Notice the line W: http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key. Try with `--secufrity-opt seccomp:unconfined` which may have security implications but does address the issue. ``` $ docker run --security-opt seccomp:unconfined -it ubuntu:22.04 ``` references: https://bugs.launchpad.net/cloud-images/+bug/1889311 https://github.com/tianon/docker-brew-ubuntu-core/issues/183 – Craig Comstock Sep 08 '22 at 14:29
  • Does this answer your question? [Why I cannot run \`apt update\` inside a fresh ubuntu:22.04?](https://stackoverflow.com/questions/71941032/why-i-cannot-run-apt-update-inside-a-fresh-ubuntu22-04) – Per Lundberg Jan 13 '23 at 13:06

4 Answers4

24

After spending half a day trying to fix the Ubuntu images (which aren't broken) I have eventually started debugging the host.

It's a docker problem. Ubuntu makes use of syscalls for better key security, which Docker didn't support yet. The solution is to update docker... or use nerdctl, runc or something similar.

Instead of apt getting the message that the syscalls aren't supported, it gets the message that permission is denied, which results in the confusing error messages.

You could technically patch ubuntu to be less secure, and to work with older docker, but that is sadly not a long term solution.

This is the PR that fixes it in the Docker(/moby) project. Note that the problem may occur in other Docker images (or other software in Ubuntu) as well, since it is ultimately caused by a change in glibc that was incompatible with the default seccomp profile of Docker.

WikiLover
  • 413
  • 2
  • 8
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Sep 16 '22 at 20:51
  • 2
    I solved it by updating docker and docker-compose . – zoro Oct 17 '22 at 05:47
  • What version of docker did you end up with when you upgraded? – iggie Oct 21 '22 at 02:04
  • 1
    I also got this bug with Docker Engine 19 and found it works on Docker Engine 20. – wearego Nov 29 '22 at 15:14
6

A very dirty but probably acceptable hack for some docker hacking is this:

apt update --allow-insecure-repositories

Which just ignores the signatures. The errors get still printed, but the package repository is updated, and you can install new packages afterwards. Even though you get warned and asked if that is really what you want to do.

ineiti
  • 324
  • 3
  • 11
  • This leads to other errors down the road. So finally I took the bullet and ran "apt update; apt upgrade" on the host system, which updated to the latest docker where this problem was solved... – ineiti Feb 26 '23 at 20:32
4

Exactly same problem

Err:3 http://security.ubuntu.com/ubuntu jammy-security InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C Reading package lists... [91mW: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.

Dockerfile (simple how to reproduce) docker-version: 18.x or 19.03/

FROM ubuntu:latest
RUN apt-get -y update

With latest Docker version 20.10.9, i did not see the issue.

some of the options tried to tshoot:

  1. sed -i 's/jammy/focal/g' /etc/apt/sources.list
  2. RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 871920D1991BC93C this fails with with another child-issue gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation

SO, solutions seems to be either update docker-version or use a tagged image where it has worked previously.

cweiske
  • 30,033
  • 14
  • 133
  • 194
TheFixer
  • 89
  • 1
  • 2
  • Same issue. If you don't want to update docker version (or you cant), you can use tagged image for `ubuntu:20.04` which will run `apt-get update` without complaints. – Steven de Salas Mar 11 '23 at 12:27
1

When you reference a docker image by name, Docker only checks to see if it exists locally or not -- it doesn't check for updates. So if there is already a version of the named image available, it will use that even though it might be stale. I suspect that's what you're seeing: some of your CI nodes must have a cached version of the image that has problems.

There are a few ways of dealing with this:

  1. Explicitly docker pull ubuntu:22.04 before calling docker run; this will pull down a newer version of the image if one exists.

  2. Add --pull always to your docker run command line. This accomplishes the same thing but without requiring an extra command execution.

  3. Reference an image by digest rather than by tag. If you use an image reference like this:

    FROM ubuntu@sha256:bace9fb0d5923a675c894d5c815da75ffe35e24970166a48a4460a48ae6e0d19

Then docker will use that exact image. You can find the image digests on docker hub.

larsks
  • 277,717
  • 41
  • 399
  • 399
  • Thanks. --pull helped me with some chaotic results I was seeing with my actual use case, which involves an intermediate base image I create myself and am frequently updating as I work out these issues. However, none of those three approaches fixed the issue. I am editing the question with my additional findings. – morningstar Jun 15 '22 at 20:37