Docker COPY with folder wildcards

Question

Given a file structure like this:

project root
|-- X.sln
|-- src
|    |-- Foo
|    |    |-- Foo.fsproj
|    |    |-- Foo.fs
|    |-- Bar
|         |-- Bar.fsproj
|         |-- Bar.fs
|-- test
     |-- Baz
          |-- Baz.fsproj

I'd like to first add all .fsproj files to my Docker image, then run a command, then add the rest of the files. I tried the following, but of course it didn't work:

COPY X.sln .
COPY **/*.fsproj .
RUN dotnet restore
COPY . .
RUN dotnet build

The idea is that after the first two COPY steps, the file tree on the image is like this:

working dir
|-- X.sln
|-- src
|    |-- Foo
|    |    |-- Foo.fsproj
|    |-- Bar
|         |-- Bar.fsproj
|-- test
     |-- Baz
          |-- Baz.fsproj

and the rest of the tree is only added in after RUN dotnet restore.

Is there a way to emulate this behavior, preferably without resorting to scripts outside of the dockerfile?

Answer 1

You can use two RUN commands to solve this problem, using the shell commands (find, sed, and xargs).

Follow the steps:

1. Find all fsproj files, with regex extract the filename without extension and with xargs use this data to create directories with mkdir;
2. Based on the previous script change the regex to create from-to syntax and use the mv command to move files to newly created folders.

Example:

    COPY *.sln ./
    COPY */*.fsproj ./
    RUN find *.fsproj | sed -e 's/.fsproj//g' | xargs mkdir
    RUN find *.fsproj | sed -r -e 's/((.+).fsproj)/.\/\1 .\/\2/g' | xargs -I % sh -c 'mv %'

References:

how to use xargs with sed in search pattern

Answer 2

If you use the dotnet command to manage your solution you can use this piece of code:

1. Copy the solution and all project files to the WORKDIR
2. List projects in the solution with dotnet sln list
3. Iterate the list of projects and move the respective *proj files into newly created directories.

COPY *.sln ./
COPY */*/*.*proj ./
RUN dotnet sln list | \
      tail -n +3 | \
      xargs -I {} sh -c \
        'target="{}"; dir="${target%/*}"; file="${target##*/}"; mkdir -p -- "$dir"; mv -- "$file" "$target"'

Answer 3

One pattern that can be used to achieve what you want without resorting to a script outside the Dockerfile is this:

    COPY <project root> .    
    RUN <command to tar/zip the directory to save a copy inside the container> \
         <command the removes all the files you don't want> \
         dotnet restore \
         <command to unpack tar/zip and restore the files> \
         <command to remove the tar/zip> \
         dotnet build

This would keep all of your operations inside the container. I've bundled them all in one RUN command to keep all of that activity into a single layer of the build. You can break them out if you need to.

Here's just one example on linux of how to recursively remove all files except the ones you don't want: https://unix.stackexchange.com/a/15701/327086. My assumption, based on your example, is that this won't be a costly operation for you.

Answer 4

Great question and I believe I have found the solution. Have .dockerignore like this

# Ignore everything except *.fsproj.
**/*.*
!**/*.fsproj

Have your Dockerfile-AA like this (please update the ls)

FROM your-image
USER root
RUN mkdir -p /home/aa
WORKDIR /home/aa
COPY . .
RUN ls
RUN ls src
RUN ls src/p1
RUN ls src/p2
RUN ls src/p3
RUN dotnet restore

Have your docker command like this

sudo docker build --rm -t my-new-img -f Dockerfile-AA .

Run it the first time, it will show only the fsproj file being copied. Run it again, you cannot see the ls result because it is using cache, great.

Obviously, you cannot have dotnet restore in the same Dockerfile, have another docker file like Dockerfile-BB

FROM my-new-img
COPY . .
RUN dotnet build

So, have your script like this

docker build --rm -t my-new-img -f Dockerfile-AA .
rm -f .dockerignore
docker build --rm -t my-new-img2 -f Dockerfile-BB .

This should work. When building my-new-img, it is going to be blazing fast. You need to try this at least 2 times, because the cache was not created right away in my past experience. This is way better than copying the project file line by line.