Powershell Copy Files with a Blacklist (Exclude) and a Whitelist (Include)

Question

I'm translating some msbuild scripts to powershell.

In msbuild, I can generate a blacklist and/or whitelist of files I want to (recursively) copy to a destination folder.

As seen below:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="AllTargetsWrapped">

    <PropertyGroup>
        <!-- Always declare some kind of "base directory" and then work off of that in the majority of cases  -->
        <WorkingCheckout>.</WorkingCheckout>
        <WindowsSystem32Directory>c:\windows\System32</WindowsSystem32Directory>
        <ArtifactDestinationFolder>$(WorkingCheckout)\ZZZArtifacts</ArtifactDestinationFolder>
    </PropertyGroup>

    <Target Name="AllTargetsWrapped">

        <CallTarget Targets="CleanArtifactFolder" />
        <CallTarget Targets="CopyFilesToArtifactFolder" />
    </Target>

    <Target Name="CleanArtifactFolder">

        <RemoveDir Directories="$(ArtifactDestinationFolder)" Condition="Exists($(ArtifactDestinationFolder))"/>
        <MakeDir Directories="$(ArtifactDestinationFolder)" Condition="!Exists($(ArtifactDestinationFolder))"/>
        <Message Text="Cleaning done" />
    </Target>

    <Target Name="CopyFilesToArtifactFolder">

        <ItemGroup>
            <MyExcludeFiles Include="$(WindowsSystem32Directory)\**\EventViewer_EventDetails.xsl" />
        </ItemGroup>

        <ItemGroup>
            <MyIncludeFiles Include="$(WindowsSystem32Directory)\**\*.xsl" Exclude="@(MyExcludeFiles)"/>
            <MyIncludeFiles Include="$(WindowsSystem32Directory)\**\*.xslt" Exclude="@(MyExcludeFiles)"/>
            <MyIncludeFiles Include="$(WindowsSystem32Directory)\**\*.png" Exclude="@(MyExcludeFiles)"/>
            <MyIncludeFiles Include="$(WindowsSystem32Directory)\**\*.jpg" Exclude="@(MyExcludeFiles)"/>            
        </ItemGroup>        

        <Copy
                SourceFiles="@(MyIncludeFiles)"
                DestinationFiles="@(MyIncludeFiles->'$(ArtifactDestinationFolder)\%(RecursiveDir)%(Filename)%(Extension)')"
        />

        </Target>

    </Project>

Can I do the same in powershell?

I have tried the below, but it creates a file called "C:\work9\MsBuildExamples\FileCopyRecursive\PowershellResults" (its a file with no extension, not a directory)

$sourceDirectory = 'c:\windows\System32\*'
$destinationDirectory = 'C:\work9\MsBuildExamples\FileCopyRecursive\PowershellResults'
$excludeFiles = @('EventViewer_EventDetails.xsl')
$includeFiles = @('*.xsl','*.xslt','*.png','*.jpg')


Copy-Item $sourceDirectory $destinationDirectory -Recurse -Include $includeFiles -Exclude $excludeFiles
# -Container:$false

APPEND:

I tried this:

$sourceDirectory = 'c:\windows\System32'
$destinationDirectory = 'C:\work9\MsBuildExamples\FileCopyRecursive\PowershellResults'
$excludeFiles = @('EventViewer_EventDetails.xsl')
$includeFiles = @('*.xsl','*.xslt','*.png','*.jpg')


Copy-Item $sourceDirectory $destinationDirectory -Recurse -Include $includeFiles -Exclude $excludeFiles

(No results, not even a file with no extension)

and I tried this:

$sourceDirectory = 'c:\windows\System32'
$destinationDirectory = 'C:\work9\MsBuildExamples\FileCopyRecursive\PowershellResults'
$excludeFiles = @('EventViewer_EventDetails.xsl')
$includeFiles = @('*.xsl','*.xslt','*.png','*.jpg')


Copy-Item $sourceDirectory $destinationDirectory -Recurse -Include $includeFiles -Exclude $excludeFiles -Container:$false

(No results, not even a file with no extension)

Answer 1

Copy-Item -Recurse, as of Windows PowerShell v5.1 / PowerShell Core 6.2.0, has its quirks and limitations; here's what I found:

^{If you have additional information or corrections, please let us know.}

There are two fundamental ways to call Copy-Item -Recurse:

• (a) specifying a directory path as the source - c:\windows\system32

• (b) using a wildcard expression as the source that resolves to multiple items in the source directory - c:\windows\system32\*

There are two fundamental problems:

• The copying behavior varies based on whether the target directory already exists - see below.

• The -Include parameter does not work properly and neither does -Exclude, though problems are much more likely to arise with -Include; see GitHub issue #8459.

DO NOT USE THE SOLUTIONS BELOW IF YOU NEED TO USE -Include - if you do need -Include, use LotPing's helpful solution.

---

Case (a) - a single directory path as the source

If the source is a single directory (or is the only directory among the items that a wildcard pattern resolved to), Copy-Item implicitly also interprets the destination as a directory.

However, if the destination directory already exists, the copied items will be placed in a subdirectory named for the source directory, which in your case means: C:\work9\MsBuildExamples\FileCopyRecursive\PowershellResults\System32

GitHub issue #2934 that - rightfully - complains about this counter-intuitive behavior

There are two basic workarounds:

If acceptable, remove the destination directory first, if it exists - which is obviously to be done with CAUTION (remove -WhatIf once you're confident that the command works as intended):

# Remove a pre-existing destination directory:
if (Test-Path $destinationDirectory) {
  Remove-Item $destinationDirectory -Recurse -WhatIf
}

# Now Copy-Item -Recurse works as intended.
# As stated, -Exclude works as intended, but -Include does NOT.
Copy-Item $sourceDirectory $destinationDirectory -Recurse

Caveat: Remove-Item -Recurse, regrettably, can intermittently act asynchronously and can even fail - for a robust alternative, see this answer.

If you want to retain a preexisting destination dir. - e.g., if you want to add to contents of the destination directory,

• Create the target dir. on demand; that is, create it only if it doesn't already exist.
• Use Copy-Item to copy the contents of the source directory to the target dir.

# Ensure that the target dir. exists.
# -Force is a quiet no-op, if it already exists.
$null = New-Item -Force -ItemType Directory -LiteralPath $destinationDirectory

# Copy the *contents* of the source directory to the target, using
# a wildcard.
# -Force ensures that *hidden items*, if any, are included too.
# As stated, -Exclude works as intended, but -Include does NOT.
Copy-Item -Force $sourceDirectory/* $destinationDirectory -Recurse

---

Case (b) - a wildcard expression as the source

Note:

• If there's exactly 1 directory among the resolved items, the same rules as in case (a) apply.

• Otherwise, the behavior is only problematic if the target item doesn't exist yet. - see below.

• Therefore, the workaround is to ensure beforehand that the destination directory exists:
  New-Item -Force -Path $destinationDirectory -ItemType Directory

If the target item (-Destination argument) doesn't exist yet:

• If there are multiple directories among the resolved items, Copy-Item copies the first directory, and then fails on the second with the following error message:
  Container cannot be copied onto existing leaf item

• If the source is a single file or resolves to files only, Copy-Item implicitly interprets a non-existent destination as a file.
  With multiple files, this means that a single destination file is created, whose content is the content of the file that happened to be copied last - i.e, there is data loss.

Answer 2

Using a suggestion I found from here (the "touch" approach)

Should Copy-Item create the destination directory structure?

I got the below to work.

I'll leave this question as "Unanswered" for a few days, maybe somebody has something better. The New-Item seems weird to me, since the Copy-Item should not fluke-out in an Include and Exclude list IMHO.

(use variable setters as per the original question)

IF (Test-Path -Path $destinationDirectory -PathType Container) 
{ 
    Write-Host "$destinationDirectory already exists" -ForegroundColor Red
} 
ELSE 
{ 
    New-Item -Path $destinationDirectory -ItemType directory 
} 


# the two scripts below are same except for "New-Item" vs "Copy-Item" , thus the "New-Item" trick-it-to-work voodoo

Get-ChildItem $sourceDirectory -Include $includeFiles -Exclude $excludeFiles -Recurse -Force |
    New-Item -ItemType File -Path  {
        Join-Path $destinationDirectory $_.FullName.Substring($sourceDirectory.Length)
    }  -Force

Get-ChildItem $sourceDirectory -Include $includeFiles -Exclude $excludeFiles -Recurse -Force |
    Copy-Item -Destination {
        Join-Path $destinationDirectory $_.FullName.Substring($sourceDirectory.Length)
    }

Answer 3

To keep an overview I shortened your vars (and changed dst to fit my env)

$src = 'c:\windows\System32'
$dst = 'Q:\Test\2017-03\15\PowershellResults'
$exc = @('EventViewer_EventDetails.xsl')
$inc = @('*.xsl','*.xslt','*.png','*.jpg')

Get-ChildItem -Path $src -Rec -Inc $inc -Exc $exc -ea silentlycontinue|ForEach{
  $DestPath = $_.DirectoryName -replace [regex]::escape($src),$dst
  If (!(Test-Path $DestPath)){MkDir $DestPath}
  $_|Copy-Item -Destination $DestPath -Force
}

Edit Essentially the same as your example, maybe a bit denser