UTF-8 BOM to UTF-8 Conversion for a large file

Question

Based on the suggestion from this thread, i have used powershell to do the UTF-8 conversion, now i am running into another problem, i have a very huge file around 18 gb which i am trying to convert on a machine with around 50GB RAM free, but this conversion process eats up all the ram and encoding fails, is there a way to limit the RAM usage or to do the conversion in chunks?

Using PowerShell to write a file in UTF-8 without the BOM

BTW below is exact code

foreach ($file in ls -name $Path\CM*.csv)
{
   $file_content = Get-Content "$Path\$file";
   [System.IO.File]::WriteAllLines("$Path\$file", $file_content);
   
   echo "encoding done : $file"

}

Answer 1

Don't Store the file's content in memory. As noted here, doing so require 3-4 times the file size in RAM. Get-Content is slow but quite memory efficient. so a simple solution may be

Get-Content -Path <FilePath> | Out-File -FilePath <FilePath> -Encoding UTF8

Note: While I haven't tried this you may want to use Add-Content instead of Out-File. The latter will sometimes reformat according to console width. Characteristic of Out-* cmdlets they traverse the for-display formatting system.

Because the content is streamed down the pipe, only one line at a time is stored in RAM. .Net memory Garbage Collection is running in the background releasing and otherwise managing RAM.

Note: [System.IO.StreamReader] and [System.IO.StreamWriter] can probably also address this issue. They may be faster, and are just as memory efficient, but they come with a syntax burden that may not be worth it, particularly if this is a one-off... That said, you can instantiate them with a System.Text.Encoding enum, so theoretically can use them for the conversion.

Answer 2

When you know that the input file is always UTF-8 with BOM, you only need to strip the first three bytes (the BOM) from the file.

Using a buffered stream, you only need to load a fraction of the file into memory.

For best performance I would use a FileStream. This is a raw binary stream and thus has the least overhead.

$streamIn = $streamOut = $null
try {
    $streamIn = [IO.FileStream]::new( $fullPathToInputFile, [IO.FileMode]::Open )
    $streamOut = [IO.FileStream]::new( $fullPathToOutputFile, [IO.FileMode]::Create )

    # Strip 3 bytes (the UTF-8 BOM) from the input file
    $null = $streamIn.Seek( 3, [IO.SeekOrigin]::Begin )

    # Copy the remaining bytes to the output file
    $streamIn.CopyTo( $streamOut )

    # You may try a custom buffer size for better performance:
    # $streamIn.CopyTo( $streamOut, 1MB )
}
finally {
    # Make sure to close the files even in case of an exception
    if( $streamIn ) { $streamIn.Close() }
    if( $streamOut ) { $streamOut.Close() }
}

You may experiment with the FileStream.CopyTo() overload that has a bufferSize parameter. In my experience, a larger buffer size (say 1 MiB) can improve performance considerably, but when it is too large, performance will suffer again because of bad cache use.

Answer 3

You can use a StreamReader and StreamWriter to do the conversion.

The StreamWriter by default outputs UTF8NoBOM.

This will take a lot of disk actions, but will be lean on memory.

Bear in mind that .Net needs full absolute paths.

$sourceFile      = 'D:\Test\Blah.txt'  # enter your own in- and output files here
$destinationFile = 'D:\Test\out.txt'

$reader = [System.IO.StreamReader]::new($sourceFile, [System.Text.Encoding]::UTF8)
$writer = [System.IO.StreamWriter]::new($destinationFile)

while ($null -ne ($line = $reader.ReadLine())) {
    $writer.WriteLine($line)
}
# clean up
$writer.Flush()
$reader.Dispose()
$writer.Dispose()

---

The above code will add a final newline to the output file. If that is unwanted, do this instead:

$sourceFile      = 'D:\Test\Blah.txt'
$destinationFile = 'D:\Test\out.txt'

$reader = [System.IO.StreamReader]::new($sourceFile, [System.Text.Encoding]::UTF8)
$writer = [System.IO.StreamWriter]::new($destinationFile)

while ($null -ne ($line = $reader.ReadLine())) {
    if ($reader.EndOfStream) {
        $writer.Write($line)
    }
    else {
        $writer.WriteLine($line)
    }
}
# clean up
$writer.Flush()
$reader.Dispose()
$writer.Dispose()