4

I came across a strange situation I try to understand.

Given the following:

entrypoint.sh (echo line was added during debugging, and resulted in discovering this)

#!/bin/bash

set -e

# echo "COMMAND: $@"
exec "$@"

Dockerfile

FROM ruby:2.4.10 AS development

RUN apt-get update && \
    apt-get install -y apt-utils && \
    apt-get install -y dumb-init && \
    apt-get install -y tzdata && \
    apt-get install -y libpq-dev

ENV LANG="C.UTF-8"
WORKDIR /app
ENTRYPOINT ["/app/entrypoint.sh"]
CMD ["bundle", "exec", "rackup", "--host", "0.0.0.0", "-p", "3000"]

FROM development
ADD Gemfile Gemfile.lock .ruby-version ./
RUN bundle config frozen true && bundle install --jobs 5 && bundle config frozen false

COPY . ./

ARG SENTRY_RELEASE=${SENTRY_RELEASE}
ENV SENTRY_RELEASE="${SENTRY_RELEASE}"

We normally build our images in a CI-pipeline, that uses the last version of the image as a cache-source to speed up things: docker build -f Dockerfile -t ${BASE_IMAGE_NAME} --build-arg="SENTRY_RELEASE=${CI_COMMIT_SHORT_SHA}" --cache-from ${CACHE_IMAGE_NAME} .

Though, now (and I think the Caching is the issue) suddenly the command changes.

When I do docker inspect $image I see:

            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "CMD [\"bundle\" \"exec\" \"rackup\" \"--host\" \"0.0.0.0\" \"-p\" \"3000\"]"
            ]

Instead of the expected (all our other images have this):

            "Cmd": [
                "bundle",
                "exec",
                "rackup",
                "--host",
                "0.0.0.0",
                "-p",
                "3000"
            ]

When I build the image without Cache (--no-cache), the command looks as expected. And makes it harder to test. This tells me it is the cache. Though as it has not yet has been a problem for us in the past, this also makes it strange.

My docker file changed before it happened, and as the base image changed in this change docker should have completely ignored the cache. I am assuming it happens when using the Cache, from the first build image after changing the base image.

Actually, this answer gives some the feeling it is expected to look like the first one. Though it is strange as it changes everything we have generated in the past, and we did not update docker in CI in-between.

When I look at docker history, the history entry to set the command actually also is correct.

<missing>      2 hours ago     /bin/sh -c #(nop)  CMD ["bundle" "exec" "rackup" "--host" "0.0.0.0" "-p" "3000"]   0B

The output of the echo command in the entrypoint.sh looks like this in the kubernetes pod: COMMAND: /bin/sh -c #(nop) CMD ["bundle" "exec" "rackup" "--host" "0.0.0.0" "-p" "3000"]

So what can cause this? and can in case the first command is correct, why is the default entrypoint not able to process it?

Thanks in advance.

Dennis van de Hoef - Xiotin
  • 1,481
  • 1
  • 13
  • 29
  • Does this answer your question? [What does #(nop) mean in docker history?](https://stackoverflow.com/questions/41298934/what-does-nop-mean-in-docker-history) – jordanvrtanoski Mar 15 '21 at 14:17
  • Sadly not it is about the meaning of NOP, where my problem is, that it adds this to the command (see the code the docker inspect returns, compared to what I expect to find). Also, the code that I find, breaks the default (as how most docker images use it) entrypoint – Dennis van de Hoef - Xiotin Mar 15 '21 at 14:18
  • Docker build is adding it as a marker that this part of the image is arriving from a lower (cached) layer that was not run during the build. Since the change was after the `CMD` line in the Dockerfile. This is why when you rerun with `--no-cache` all layers will be run, so none of them will have the `#(nop)` mark. – jordanvrtanoski Mar 15 '21 at 14:24
  • Do you have the output from the `echo` from the entry point? – jordanvrtanoski Mar 15 '21 at 14:27
  • Also, what is the version you have, I can not reproduce it on`20.10.3`. My `Cmd` is untouched, I see the `#(nop)` in history, but the `Cmd` is as expected. – jordanvrtanoski Mar 15 '21 at 14:37
  • 1
    I added the Echo output to the question. Our CI runs Docker version 20.10.2, though I have only seen this for this one image, and we build 10+ images a day the same way without issues. – Dennis van de Hoef - Xiotin Mar 15 '21 at 14:50
  • I found couple of errors which are similar to this in github. Here is link to one https://github.com/docker/for-win/issues/5269. But I am not able to reproduce it on my version, which is even more strange. It looks like it was fixed but I was not able to find the fix #. Can you try on an a higher version of docker if you can reproduce? – jordanvrtanoski Mar 15 '21 at 15:16
  • Sorry, I can't update CI now :( Also reproducing is hard, as after manny tests it is hard to find the image that broke it at first – Dennis van de Hoef - Xiotin Mar 17 '21 at 07:57
  • Can you download and tag the bad image to your local machine and reproduce there? – NobodysNightmare Mar 17 '21 at 08:12
  • Are you using buildx or legacy build? – jordanvrtanoski Mar 17 '21 at 08:49
  • 1
    It looks to me like the issue is related with the `--cache-from` flag. If the cache-from containers is fetched from the repository, the layers from the cache will all be `#(nop) `, so this is the time when the `CMD` becomes corrupted. – jordanvrtanoski Mar 17 '21 at 09:11

0 Answers0