35

I run a build on a node application and then use the artifacts to build a docker image. The COPY command that moves my source in place isn't detecting changes to the source files after a build; its just using the cache.

Step 9/12 : COPY server /home/nodejs/app/server ---> Using cache ---> bee2f9334952

Am I doing something wrong with COPY or is there a way to not cache a particular step?

Jason Leach
  • 3,889
  • 7
  • 37
  • 54

8 Answers8

21

I found this in the Docker documentation:

For the ADD and COPY instructions, the contents of the file(s) in the image are examined and a checksum is calculated for each file. The last-modified and last-accessed times of the file(s) are not considered in these checksums. During the cache lookup, the checksum is compared against the checksum in the existing images. If anything has changed in the file(s), such as the contents and metadata, then the cache is invalidated.

So, as far as I understand, the cache should be invalidated. You can use the --no-cache command-line option to make sure. If you get the correct behavior with --no-cache and an incorrect behavior without it, you would have discovered a bug and should report it.

ib.
  • 27,830
  • 11
  • 80
  • 100
herm
  • 14,613
  • 7
  • 41
  • 62
4

This was interesting. I found out that COPY WAS working, it just looked like it wasn't.

I was rebuilding the images and restarting my containers, but the container was still using the old image. I had to remove my containers, and then when I started them up they used the newer image that was created, and I could see my changes.

Here is another thread that deals with this more accurately diagnosed (in my case).

Nick Brady
  • 6,084
  • 1
  • 46
  • 71
  • 1
    Thanks so much for this post! I was experiencing the same lack of attention and failed to notice that the container's image id was different from the id of the latest image I've just built. – Danny Apostolov Nov 10 '21 at 13:37
  • thanks a lot for this feedback!! It is amazing how you could figure it out – debuti Jul 12 '23 at 22:28
1

For me, the problem was in my interpretation of Docker build output. I did not realize not only the last version of a layer is cached, but also all previous ones.

I was testing cache invalidation by changing a single file back and forth. After the first change, the cache was invalidated OK, but after changing back, the layer was taken from cache, which seemed as if the invalidation logic did not work properly.

https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#leverage-build-cache

Palec
  • 12,743
  • 8
  • 69
  • 138
1

It is likely a bug, but hard to replicate. It happens to me in Jenkins builds when I copy a new file to existing folder that used to be copied its entirety using single Dockerfile COPY command. To make cache invalidation work correctly (and avoid rebuilding earlier layers as --no-cache would), it is necessary to run docker build --tag <REPO>/<IMAGE> . on the host (outside of Jenkins).

mirekphd
  • 4,799
  • 3
  • 38
  • 59
0

You could try with ADD instead. It will invalidate the cache for the copy. The bad side is that it will also invalidate the cache for the other commands after it. If your ADD is in the last steps it shouldn't impact to much the build process.

Note: The first encountered ADD instruction will invalidate the cache for all following instructions from the Dockerfile if the contents of have changed. This includes invalidating the cache for RUN instructions. See the Dockerfile Best Practices guide for more information. https://docs.docker.com/engine/reference/builder/#add

Guillaume
  • 423
  • 1
  • 4
  • 11
  • This is great info also. – Jason Leach Mar 28 '17 at 15:53
  • 1
    The same is valid for COPY as well: "The first encountered COPY instruction will invalidate the cache for all following instructions from the Dockerfile if the contents of have changed. This includes invalidating the cache for RUN instructions. See the Dockerfile Best Practices guide – Leverage build cache for more information." -- https://docs.docker.com/engine/reference/builder/#copy – Danny Apostolov Nov 10 '21 at 13:54
0

Had the same issue. After considering @Nick Brady's post (thanks for the suggestion!), here is my current update procedure that seems to be working fine:

svn update --non-interactive --no-auth-cache --username UUU --password PPP
docker build . -f deploy/Dockerfile -t myimage
docker stop mycontainer
docker rm mycontainer
docker run --name=mycontainer -p 80:3100 -d --restart=always \
    --env-file=deploy/.env.production myimage

The magic here is to not simply restart the container (docker restart mycontainer), as this would actually stop and run again the old container that was instantiated from a previous version of myimage. Stopping and destroying the old container and running a new one instead results in a fresh container instantiated from the newly built myimage.

Danny Apostolov
  • 479
  • 3
  • 8
0

for me changing from COPY to ADD did the trick, and now the image is built from this line when the added file changed.

lior.mor
  • 158
  • 1
  • 9
-5

From the point of view of Docker this is just like any other command.

Docker sees that this line didn't change, so it caches it.

Similarly if you have a curl command in your Dockerfile, Docker doesn't fetch the URL just to change if it changed. It checks if the command changed or not, not it's result.

rsp
  • 107,747
  • 29
  • 201
  • 177