How to do a copy /b in a powershell script to insert a BOM marker, but as a batch for files that match a filter and changes the ext on output?

Question

REASONS WHY THIS IS NOT A DUPLICATE

Since 3 people have already voted to close, I guess I should explain why this question is not a duplicate:

I cannot use cat or >> as these mess up the encoding of the files, which are UTF8 on input and need to be UTF8-BOM on output.
The linked question does not show how to loop through all files that match a given pattern in a directory, and concatenate a single file to each of the matching files on output, plus give the new file a different extension.
Using Set-Content is not Powershell 6 future-proof, since Set-Content will NOT add a BOM marker. In Powershell 5 and below, it sometimes adds a BOM marker and sometimes not, depending on the configuration settings of the executing user. See 'quick note on encoding' at the end of this article.

So in conclusion I am looking for a solution that uses copy (hence the question title) and does NOT use Cat or Set-Content.

I need to loop through certain files in a given directory and run the following on each file:

copy /b BOMMarker.txt+InputFile.dat OutputFile.txt

This inserts the contents of the BOMMarker.txt file at the start of the InputFile.dat and writes the output to OutputFile.txt

I found this question which explains how I can loop through the folder to load each file into Powershell, but how do I apply the "copy /b" command so that I can get the BOM marker at the start of each file?

EDIT

The comment from Jeroen indicates I can just do Set-Content on the output file, as Powershell will automatically add the BOM at the start.

But I also need to change the extension. So the output filename needs to be the same as the input filename, just with a changed extension (from .dat to .txt) and including the BOM.

I am guessing I can use Path.ChangeExtension somehow to do this, but not sure how to combine that with also adding the BOM.

EDIT - for Bounty

The example answer I posted does not work in all environments I tested it, and I do not know why (possibly different default Powershell setttings) but also, it is not future proof since Powershell 6 will not output BOM by default.

From the given directory, I need to process all files that match the filter (DIL_BG_TXN*.dat).

For each of those files, I need to copy it with a BOM at the start but the resultant new file needs to be the same name but with the extension .txt instead of .dat.

Possible duplicate of [How do I concatenate two text files in PowerShell?](https://stackoverflow.com/questions/8749929/how-do-i-concatenate-two-text-files-in-powershell) — Malcolm McCaffery, Jan 09 '18 at 12:19
If you want to add a BOM marker to each file, rather than concatenate files like this, you may want to use `Set-Content OutputFile.txt (Get-Content InputFile.txt) -Encoding UTF8` (or `UTF16`). PowerShell happens to add the BOM by default, so you don't need the usual contortions of `cmd`. — Jeroen Mostert, Jan 09 '18 at 12:27
@JeroenMostert Thanks, please post that as the answer and I will accept it. — rmcsharry, Jan 09 '18 at 12:50
*"But I also need to change the extension."* Probably just `Set-Content OutputFile.txt (Get-Content InputFile.dat) -Encoding UTF8` — Mike Sherrill 'Cat Recall', Jan 09 '18 at 20:58

FstTesla · Accepted Answer · 2018-01-26T15:24:14.440

This solutions uses streams, that reliably read and write as-is:

$bomStream = [IO.File]::OpenRead('BOMMarker.txt')
$location = "" # set this to the folder location

$items = Get-ChildItem -Path $location -Filter DIL_BG_TXN*.dat

foreach ($item in $items) {
    $sourceStream = [IO.File]::OpenRead($item.FullName)
    $targetStream = [IO.File]::OpenWrite([IO.Path]::ChangeExtension($item.FullName, '.txt'))
    $bomStream.CopyTo($targetStream)
    $sourceStream.CopyTo($targetStream)
    $targetStream.Flush()
    $targetStream.Close()
    $sourceStream.Close()
    $bomStream.Position = 0
}
$bomStream.Close()

Of course please write the absolute path of BOMMarker.txt (1st line) according to its location.

score 2 · Answer 2 · answered Jan 22 '18 at 14:56

This finally worked:

$Location = "C:\Code\Bulgaria_Test"

$items = Get-ChildItem -Path $Location -Filter DIL_BG_TXN*.dat

ForEach ($item in $items) {

    Write-Host "Processing file - " $item

    cmd /c copy /b BOMMarker.txt+$item ($item.BaseName + '.txt')

}

Description:

Set the directory location where all the .dat files are.
Load only those files that match the filter into the array $items.
Loop through each $item in the array.
With each $item, call cmd shell with the copy /b command and concatenate the bom marker file with the $item file and write the result to the basename of $item plus the new extension.

How to do a copy /b in a powershell script to insert a BOM marker, but as a batch for files that match a filter and changes the ext on output?

2 Answers2