1

What is the fastest way to prepend header to a file? 100 megabytes file took about 10 minutes to prepend header to a file. In java it take two seconds.

The example below is very slow:

$header = 'address;name'
Import-Csv .\myfile.csv -Header $header

or

Add-Content -path .\NewTest.csv -value "address;name"
$contents = Get-Content .\Test.csv
Add-Content -path .\NewTest.csv -value $contents
marsze
  • 15,079
  • 5
  • 45
  • 61
gavin newsom
  • 11
  • 4
  • If you want to know which one is faster, why don't you try both and measure how long it takes? [`Measure-Command` is your friend](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/measure-command) – Mathias R. Jessen Oct 28 '20 at 14:07
  • If you want to do this using Import-Csv, you need to specify your csv uses the semi-colon as delimiter character using `-Delimiter ';'` – Theo Oct 28 '20 at 14:09
  • PowerShell differs from Java where it is optimized for streaming (pipeline usage). Generally, try to avoid to load the whole table into memory (do not assign it to a variable). Instead, directly pipe each object to the next cmdlet and eventually unload it from memory: `import-csv .\myfile.csv | foreach-object {` do your thing for each item`} | export-csv .\output.csv` (*please add more details of what you try to achieve*) – iRon Oct 28 '20 at 14:22
  • Bottom line, you can't justify the performance of PowerShell by taking the sum of some individual commands, [*as the performance of a complete (PowerShell) solution is supposed to be better than the sum of its parts*](https://stackoverflow.com/a/59437162/1701026). In other words, it depends on what you want to do with each item with could potentially take longer than the overhead of loading and unloading a single item, see [Advocating native PowerShell](https://stackoverflow.com/a/58357033/1701026) – iRon Oct 28 '20 at 15:32

2 Answers2

1

How about

Set-Content -Path 'D:\Test\NewTest.csv' -Value "address;name"
[System.IO.File]::AppendAllText('D:\Test\NewTest.csv', [System.IO.File]::ReadAllText('D:\Test\Test.csv'))

You need to use full absolute paths for this

Theo
  • 57,719
  • 8
  • 24
  • 41
0

Using .NET methods for working with file contents will usually be faster than the PowerShell cmdlets, because they have a lot of overhead etc. (Look at this performance comparison)

Here's a short and simple version:

$path = (Get-Item .\myfile.csv).FullName
$contents = [System.IO.File]::ReadAllText($path)
Set-Content -Path $path -Value "address;name"
[System.IO.File]::AppendAllText($path, $contents)

Here is another possible example, which will process one line at a time, and not read the entire text at once, so it's easier on your memory.

# your original file
$instream = New-Object System.IO.StreamReader "c:\test\myfile.csv"
# new temporary file
$outstream = New-Object System.IO.StreamWriter "c:\test\myfile2.csv"
# write the header
$outstream.WriteLine("address;name")
# write remaining contents
foreach ($line in $instream.ReadLine()) {
    $outstream.WriteLine($line)
}
$instream.Dispose()
$outstream.Dispose()

# replace the old file
Remove-Item "c:\test\myfile.csv"
Rename-Item "c:\test\myfile2.csv" "myfile.csv"
marsze
  • 15,079
  • 5
  • 45
  • 61