best way to combine extra large text files

Question

I have several text files that I would like to combine together into one. Size of folder containg all the separate text files is nearly 8Gb. I tried the following in powershell :

cat example*.txt | sc allexamples.txt

but as soon as the size of combined file reaches close to 800 Mb the laptop is hanging and Im not able to proceed further. Any other way in which I can do this?

I would go for a LibreOffice `.odt` document as it is compressed. But probably programming required. — Joop Eggen, Aug 04 '16 at 09:45
Have you tried (from command prompt) with `copy /b example*.txt allexamples.txt` ? — MC ND, Aug 04 '16 at 09:50
Asside from *how*, *why*? What does one large unmanagable 8GB file gain you you think you can't do from seperate files? — Lieven Keersmaekers, Aug 04 '16 at 10:08

score 5 · Answer 1 · answered Aug 04 '16 at 09:48

It can be done in PowerShell as well, you just need to break it up a bit or use a stream writer which is even less work so you're not hanging onto things in memory.

$file = [system.io.file]::OpenWrite("$($pwd.Path)\allexamples.txt")
$writer = New-Object System.IO.StreamWriter($file)

cat example*.txt | ForEach-Object { $writer.WriteLine($_) }

$writer.Close()
$file.Close()

score 0 · Answer 2 · answered Aug 04 '16 at 09:42

The reason why it hangs is due to the lack of memory. what your command essentially does is piping the output to the buffer and after that redirected to another file. what you should do is write a simple script in python to merge the file for you. Python has functions/classes that supports such operation.

score 0 · Answer 3 · answered Aug 04 '16 at 11:47

Get-Content -readcount can help a lot here to find the righ Balance between Memory consumption and speed. I'd try something like

Foreach ($File in (Get-Childitem <Folder> -Filter example*.txt)) {
    Get-Content $File -Readcount 1000 | Out-File allexamples.txt -Append
}

What the best value for readcount is depends on the filesize, Memory available and processing power.

best way to combine extra large text files

3 Answers3

Linked