How to include files outside of build context without specifying a different Dockerfile path?

Question

This is basically a follow-up question to How to include files outside of Docker's build context?: I'm using large files in all of my projects (several GBs) which I keep on an external drive, only used for development.

I want to COPY or ADD these files to my docker container when building it. The answer linked above allows one to specify a different path to a Dockerfile, potentially extending the build context. I find this unpractical, since this would require setting the build context to system root (?), to be able to include a single file.

Long story short: Is there any way or workaround to include a file that is far removed from the docker build context?

Answer 1

Three suggestions on things you could try:

include a file that is far removed from the docker build context?

You could construct your own build context by cp (or tar) files on the host into a dedicated directory tree. You don't have to use the actual source tree or your build tree.

rm -rf docker-build
mkdir docker-build
cp -a Dockerfile build/the-binary docker-build
cp -a /mnt/external/support docker-build
docker build ./docker-build
# reads docker-build/Dockerfile, and the files in the
# docker-build directory, but nothing else; only sends
# the docker-build directory to Docker as the build context

large files [...] (several GBs)

Docker doesn't deal well with build contexts this large. In the past I've at least seen docker build take a long time just on the step of sending the build context to itself, and docker push and docker pull have network issues when trying to send the gigabyte+ layer around.

It's a little hacky and breaks the "self-contained image" model a little bit, but you can provide these files as a Docker bind-mount instead of including them in the image. Your application needs to know what to do if the data isn't there. When you go to deploy the application, you also need to separately distribute the files alongside the Docker image and other deployment artifacts.

docker run \
  -v /mnt/external/support:/app/support
  ...
  the-image-without-the-support-files

only used for development

Potentially you can get away with not using Docker at all during this phase of development. Use a local source tree and local development tools; run your unit tests against these large test fixtures as needed. Build a Docker image only when you're about to run pre-commit integration tests; that may be late enough in the development cycle that you don't need these files.

Answer 2

I think the main thing you are worried about is that you do not want to send all files of a directory to docker daemon while it builds the image. When directory was so big (in GBss) it takes lot of time to build an image.

If the requirement is to just use those files while you build anything inside docker, you can mount those to the container.

A tricky way

1. Run a container with base image and mount the direcotries inside it. docker run -d -v local-path:container-path

2. Get inside the container docker exec -it CONTAINER_ID bash

3. Run build step ./build-something.sh

4. Create image from the running container docker commit CONTAINER_ID

5. Tag the image docker tag IMAGE_ID tag:v1. You can get Image ID from previous command

From long term perspective this method may seem to be very tedious, but if you want to build image for 1 or 2 times , you can try this method.

I tried this for one of my docker image, as I want to avoid large amount of files sent to docker daemon during image build

Answer 3

The copy command gets source and destination values, just specify full absolute path to your hard drive mount point as the src directory

COPY /absolute_path/to/harddrive /container/path