Speed up Out-File across multiple items

Question

I'm new to PowerShell but I'm trying to work through a folder of multiple files. I have files that will randomly be UTF-8, UTF-8 BOM, LE, BE, anything. I need all of them to be UTF-8 to be ingested by my ETL software our company uses. I tried numerous methods and the one below is the only one I got consistently working for any encoding.

This works almost instantly for one record, but when applying the foreach it takes roughly a minute and half for three items. I'm going to run this daily across thousands so this is unreasonable.

Any help would be appreciated.

'''

# Primary Variables
$Path = ".\Test\"
$Filename = "DriveThru_FileList.txt"
$Files = Get-ChildItem $Path\*.txt -Exclude ($Filename)


Foreach ($File in $Files){
    $Content = Get-Content $File
    $Content | Out-File -FilePath $File -Encoding "Default"
}

'''

[Here](https://stackoverflow.com/questions/28181000/improve-powershell-performance-to-generate-a-random-file) is an answer with a set of tips to improve performance over `Out-File` which was designed for usability and not speed. In particular, I would try switching to `Set-Content` and use `Measure-Command` as described. — Rich Moss, Dec 13 '19 at 00:10
Wouldn't that be `| out-file -encoding utf8` ? You can try `set-content -value $content -encoding utf8`. — js2010, Dec 13 '19 at 19:37

Theo · Accepted Answer · 2019-12-13T15:53:40.350

I think you can speed this up by using .NET methods:

$Path     = ".\Test\*"
$Filename = "DriveThru_FileList.txt"
# just store the files FullName in a string array
# when using te -Exclude switch, you must either also use -Recurse or
# have the path end in '\*' like in this example
$Files    = Get-ChildItem -Path $Path -Filter '*.txt' -Exclude $Filename | Select-Object -ExpandProperty FullName

foreach ($file in $Files) {
    # by default, this will create files in UTF-8 encoding without a Byte-Order Mark
    # you can add a 3rd parameter if you need a different encoding like
    # [System.IO.File]::WriteAllText($file, ([System.IO.File]::ReadAllText($file)), [System.Text.Encoding]::UTF8)

    [System.IO.File]::WriteAllText($file, ([System.IO.File]::ReadAllText($file)))
}

This worked perfectly. I'm interested to see how long it takes against my whole dataset but that shortened the three to no more than 10 seconds. I greatly appreciate it! — CK_Dev_Justin, Dec 13 '19 at 18:17

Speed up Out-File across multiple items

1 Answers1