How to ignore folders to send in docker build context

Question

I am facing an issue of large docker build context because of my project structure. In my root directory I have lib folder for common code and folders of micro-services. Now I want to build for miscroservice1 to include only lib folder and to ignore other microservices.

I am running docker build command in root folder because running command in microservice folder giving error Forbidden path outside the build context

rootFolder
-- lib
-- microservice1/Dockerfile
-- microservice2/Dockerfile
-- microservice3/Dockerfile

I have two solutions but didn't try for now

To add symlinks for lib in my microservice folder
To write script for each docker build to add lib folder in specific microservice folder and then run docker build.

I am trying the above two solutions. Can anyone suggest any best practice?

.dockerignore will not resolve this because I need to include specific folders based on build for specific microservice and in my case I need to add .dockerignore in root folder and that should be only one — Ibtesam Latif, Aug 31 '19 at 18:17

score 54 · Answer 1 · edited Jun 20 '20 at 09:12

54

You can create .dockerignore in your root directory and add

microservice1/
microservice2/
microservice3/

to it, just like .gitignore does during tracking files, docker will ignore these folders/files during the build.

Update

You can include docker-compose.yml file in your root directory, look at docker-compose for all the options, such as setting environment, running a specific command, etc, that you can use during the build process.

version: "3"
services:
  microservice1:
    build:
      context: .
      dockerfile: ./microservice1/Dockerfile
    volumes:
      - "./path/to/share:/path/to/mount/on/container"
    ports:
      - "<host>:<container>"
    links:
      - rootservice # defines a dns record in /etc/hosts to point to rootservice
  microservice2:
    build:
      context: .
      dockerfile: ./microservice2/Dockerfile
    volumes:
      - "./path/to/share:/path/to/mount/on/container"
    ports:
      - "<host>:<container>"
    links:
      - rootservice # defines a dns record in /etc/hosts to point to rootservice
      - microservice1
  rootservice:
    build:
      context: .
      dockerfile: ./Dockerfile
    volumes:
      - "./path/to/share:/path/to/mount/on/container"
    ports:
      - "<host>:<container>"
    depends_on:
      - microservice1
      - microservice2
    ports:
      - "<host1>:<container1>"
      - "<host2>:<container2>"

This will be your build recipe for your microservices, you can now run docker-compose build to build all your images.

edited Jun 20 '20 at 09:12

Community

1
1

answered Aug 31 '19 at 18:19

Minato

4,383
1
22
28

When I will run build for microservice1. Do this folder will include in build? – Ibtesam Latif Aug 31 '19 at 18:21
since you have `Dockerfile` for each of the service you'll have to build each service separately. – Minato Aug 31 '19 at 18:25
If you know the `microservice` architecture, which says every microservice has to be its separate image, and all the services communicate over IP protocol, which means that each service has to be built separately and deployed separately. if you want to build all you services using a single command or define relation between different services you might wanna look at [docker-compose](https://docs.docker.com/compose/) – Minato Aug 31 '19 at 18:28
but .dockerignore will be one for all builds – Ibtesam Latif Aug 31 '19 at 18:29
@IbtesamLatif not really, each docker workspace can have their own `.dockerignore`, which means your microservice1 can have its own `.dockerignore` – Minato Aug 31 '19 at 18:30
Where I need to run docker build command for microservice1. In root folder or in microservice1 folder? – Ibtesam Latif Aug 31 '19 at 18:32
@IbtesamLatif I think you need a mix of `.dockerignore` and `docker-compose` solution. – Minato Aug 31 '19 at 18:32
@IbtesamLatif I have updated my answer, to include the solution that I was proposing, this is to circumvent the script that you wanted to write. – Minato Aug 31 '19 at 18:46
I have fixed it by adding my custom script that will run in my docker build to include required folders – Ibtesam Latif Sep 30 '19 at 14:51
Unfortunately it is suboptimal because the context will be shared for all microservices, which leads to cache misses if other microservice change – Jairo Andres Velasco Romero Jun 05 '22 at 01:44

score 5 · Answer 2 · answered Aug 31 '19 at 21:03

If the only tool you have is Docker, there aren't very many choices. The key problem is that there is only one .dockerignore file. That means you always have to use your project root directory as the Docker context directory (including every services' sources), but you can tell Docker which specific Dockerfile within that to use. (Note that all COPY directives will be relative to the rootFolder in this case.)

docker build rootFolder -f microservice1/Dockerfile -t micro/service1:20190831.01

In many languages there is a way to package up the library (C .a, .h, and .so files; Java .jar files; Python wheels; ...). If your language supports that, another option is to build the library, then copy (not symlink) the library into each service's build tree. Using Python's wheel format as an example:

pip wheel ./lib
cp microlib.whl microservice1
docker build microservice1 -t micro/service1:20190831.01
# Dockerfile needs to
# RUN pip install ./microlib.whl

Another useful variant on this is a manual multi-stage build. You can have lib/Dockerfile pick some base image, and then install the library into that base image. Then each service's Dockerfile starts FROM the library image, and has it preinstalled. Using a C library as an example:

# I am lib/Dockerfile

# Build stage
FROM ubuntu:18.04 AS build
RUN apt-get update && apt-get install build-essential
WORKDIR /src
COPY ./ ./
RUN ./configure --prefix=/usr/local && make
# This is a typical pattern implemented by GNU Autoconf:
# it actually writes files into /src/out/usr/local/...
RUN make install DESTDIR=/src/out

# Install stage -- service images are based on this
FROM ubuntu:18.04
COPY --from=build /src/out /
RUN ldconfig

# I am microservice1/Dockerfile
ARG VERSION=latest
FROM micro/lib:${VERSION}
# From the base image, there are already
# /usr/local/include/microlib.h and /usr/local/lib/libmicro.so
COPY ...
RUN gcc ... -lmicro
CMD ...

There is also usually an option (again, depending on your language and its packaging system) to upload your built library to some server, possibly one you're running yourself. (A Python pip requirements.txt file can contain an arbitrary HTTP URL for a wheel, for example.) If you do this then you can just declare your library as an ordinary dependency, and this problem goes away.

Which of these works better for you depends on your language and runtime, and how much automation of multiple coordinated docker build commands you're willing to do.

How to ignore folders to send in docker build context

2 Answers2

Update