0

I Have large .txt file, over 10gb I should open this file and make simple editions: - remove duplicate lines - split this file (like 5x2gb files)

What software I need? Windows pref... But if not possible Linux can be great too. Thanks!

Verciak Nostale
  • 35
  • 1
  • 1

4 Answers4

3
    EmEditor

On Windows, you can install an application called "EmEditor".

This application can open large files. It is a licensed application however you can use it for a one month trial period. Make the change and uninstall.

I had faced this issue where I couldn't edit a large 4GB .sql file. This tool editor was the only editor which could get the job done.

Mashmoom
  • 487
  • 4
  • 10
0

No sure in Windows, but in Linux you can use combinations of grep, awk and sed. Try finding them in SO. You can refer to the query below. Basic grep/sed/awk script to find duplicates.

As for split, you can try and use How to split one text file into multiple *.txt files?

Soumya
  • 1,350
  • 3
  • 19
  • 34
0

You can achieve this using Vim or Emacs

I would recommend using Vim since it's packed with Git bash

You quit Vim with :q and if the buffer changed you can quit without saving with :q!

If Vim if not for you, check this answer and this link, maybe the editors sugested in those links are better suited for you.

Vim CheatSheet

Removing duplicated lines using Vim

Split file using Vim (vi.stackexchange.com)

Cheloide
  • 793
  • 6
  • 21
0

I will recommend you to use 010 editor. I have edited file of 40GB. It works like a charm.

Ankit Chauhan
  • 384
  • 2
  • 18