Is there a more elegant way to copy specific files using Docker COPY to the working directory?

Question

Attempting to create a container with microsoft/dotnet:2.1-aspnetcore-runtime. The .net core solution file has multiple projects nested underneath the solution, each with it's own .csproj file. I am attemping to create a more elegant COPY instruction for the sub-projects

The sample available here https://github.com/dotnet/dotnet-docker/tree/master/samples/aspnetapp has a solution file with only one .csproj so creates the Dockerfile thusly:

COPY *.sln .
COPY aspnetapp/*.csproj ./aspnetapp/
RUN dotnet restore

It works this way

COPY my_solution_folder/*.sln .
COPY my_solution_folder/project/*.csproj my_solution_folder/
COPY my_solution_folder/subproject_one/*.csproj subproject_one/
COPY my_solution_folder/subproject_two/*.csproj subproject_two/
COPY my_solution_folder/subproject_three/*.csproj subproject_three/

for a solution folder structure of:

my_solution_folder\my_solution.sln
my_solution_folder\project\my_solution.csproj
my_solution_folder\subproject_one\subproject_one.csproj
my_solution_folder\subproject_two\subproject_two.csproj
my_solution_folder\subproject_three\subproject_three.csproj

but this doesn't (was a random guess)

COPY my_solution_folder/*/*.csproj working_dir_folder/*/

Is there a more elegant solution?

Answer 1

2021: with BuildKit, see ".NET package restore in Docker cached separately from build" from Palec.

---

2018: Considering that wildcard are not well-supported by COPY (moby issue 15858), you can:

• either experiment with adding .dockerignore files in the folder you don't want to copy (while excluding folders you do want): it is cumbersome
• or, as shown here, make a tar of all the folders you want

Here is an example, to be adapted in your case:

find .. -name '*.csproj' -o -name 'Finomial.InternalServicesCore.sln' -o -name 'nuget.config' \
  | sort | tar cf dotnet-restore.tar -T - 2> /dev/null

With a Dockerfile including:

ADD docker/dotnet-restore.tar ./

The idea is: the archive gets automatically expanded with ADD.

---

The OP sturmstrike mentions in the comments "Optimising ASP.NET Core apps in Docker - avoiding manually copying csproj files (Part 2)" from Andrew Lock "Sock"

The alternative solution actually uses the wildcard technique I previously dismissed, but with some assumptions about your project structure, a two-stage approach, and a bit of clever bash-work to work around the wildcard limitations.

We take the flat list of csproj files, and move them back to their correct location, nested inside sub-folders of src.

# Copy the main source project files
COPY src/*/*.csproj ./  
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done

---

L01nl suggests in the comments an alternative approach that doesn't require compression: "Optimising ASP.NET Core apps in Docker - avoiding manually copying csproj files", from Andrew Lock "Sock".

FROM microsoft/aspnetcore-build:2.0.6-2.1.101 AS builder
WORKDIR /sln

COPY ./*.sln ./NuGet.config  ./

# Copy the main source project files
COPY src/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done

# Copy the test project files
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done

RUN dotnet restore

# Remainder of build process

This solution is much cleaner than my previous tar-based effort, as it doesn't require any external scripting, just standard docker COPY and RUN commands.
It gets around the wildcard issue by copying across csproj files in the src directory first, moving them to their correct location, and then copying across the test project files.

Answer 2

One other option to consider is using a multi-stage build to prefilter / prep the desired files. This is mentioned on the same moby issue 15858.

For those building on .NET Framework, you can take it a step further and leverage robocopy.

For example:

FROM mcr.microsoft.com/dotnet/framework/sdk:4.8 AS prep

# Gather only artifacts necessary for NuGet restore, retaining directory structure
COPY / /temp/
RUN Invoke-Expression 'robocopy C:/temp C:/nuget /s /ndl /njh /njs *.sln nuget.config *.csproj packages.config'

[...]

# New build stage, independent cache
FROM mcr.microsoft.com/dotnet/framework/sdk:4.8 AS build

# Copy prepped NuGet artifacts, and restore as distinct layer
COPY --from=prep ./nuget ./
RUN nuget restore

# Copy everything else, build, etc
COPY src/ ./src/
RUN msbuild

[...]

The big advantage here is that there are no assumptions made about the structure of your solution. The robocopy '/s' flag will preserve any directory structure for you.

Note the '/ndl /njh /njs' flags are there just to cut down on log noise.

Answer 3

Qnother solution, maybe a bit slower but all in one

Everything in one file and one command docker build .

I've split my Dockerfile in 2 steps,

• First image to tar the *.csproj files
• Second image use the tar and setup project

code:

FROM ubuntu:18.04 as tar_files
WORKDIR /tar
COPY . .
RUN find . -name "*.csproj" -print0 | tar -cvf projectfiles.tar --null -T -


FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build
WORKDIR /source

# copy sln
COPY *.sln .

# Copy all the csproj files from previous image
COPY --from=tar_files /tar/projectfiles.tar .
RUN tar -xvf projectfiles.tar
RUN rm projectfiles.tar

RUN dotnet restore

# Remainder of build process

Answer 4

In addition to VonC's answer (which is correct), I am building from a Windows 10 OS and targetting Linux containers. The equivalent to the above answer using Windows and 7z (which I normally have installed anyway) is:

7z a -r -ttar my_project_files.tar .\*.csproj .\*.sln .\*nuget.config

followed by the ADD in the Dockerfile to decompress.

Be aware that after installing 7-zip, you will need to add the installation folder to your environment path to call it in the above fashion.

Looking at the moby issue 15858, you will see the execution of the BASH script to generate the tar file and then the subsequent execution of the Dockerfile using ADD to extract.

Fully automate either with a batch or use the Powershell execution as given in the below example.

Pass PowerShell variables to Docker commands

Answer 5

I use this script

COPY SolutionName.sln SolutionName.sln
COPY src/*/*.csproj ./
COPY tests/*/*.csproj ./
RUN cat SolutionName.sln \
| grep "\.csproj" \
| awk '{print $4}' \
| sed -e 's/[",]//g' \
| sed 's#\\#/#g' \
| xargs -I {} sh -c 'mkdir -p $(dirname {}) && mv $(basename {}) $(dirname {})/'

RUN dotnet restore "/src/Service/Service.csproj"

COPY ./src ./src
COPY ./tests ./tests
RUN dotnet build "/src/Service/Service.csproj" -c Release -o /app/build

1. Copy solution file
2. Copy project files
3. (optional) Copy test project files
4. Make linux magic (scan sln-file for projects and restore directory structure)
5. Restore packages for service project
6. Copy sources
7. (optional) Copy test sources
8. Build service project

This is working for all Linux containers