how to handle large text files?

Question

I have a file which contains around 10 million rows and size is around 400mb , my system is just not able to handle it . when i try opening the file using gedit , it just gets stuck . Is there any way to handle such large files of data.

Assuming that you don't want to write a program to do that, this might help: http://stackoverflow.com/questions/159521/text-editor-to-open-big-giant-huge-large-text-files Does this belong on superuser? — weiqure, Oct 18 '10 at 18:31
@DOK: Agreed, but to be fair, there are only five other questions of which he's accepted an answer on one. (The seventh is this one, and one of the other six has been closed.) @Rahul: 20% still isn't good and suggests either that your questions are not sufficiently detailed or answerable, or that people *have* answered them but you haven't marked an answer. — T.J. Crowder, Oct 18 '10 at 18:35
@DOK :ohh sorry .. i would definitely do that . i apologise since i am new to the committee, it will take a while being accustomed to it . — Rahul, Oct 18 '10 at 18:36
@user67011 well vim is doing great . should have used that first. — Rahul, Oct 18 '10 at 18:50

score 2 · Answer 1 · answered Jun 19 '13 at 16:05

2

A web based great utility that can handle large file is Google Refine Tool.

You can find https://code.google.com/p/google-refine/wiki/FaqAllocateMoreMemory [details for memory allocation]

answered Jun 19 '13 at 16:05

Raghvendra

216
3
17

The OP mentions `gedit`, so they just want to look at the content of a text file, not transform it. – Raedwald Jun 19 '13 at 16:26

score 1 · Answer 2 · answered Oct 18 '10 at 18:56

1

A great utility for Windows is Large Text Viewer

I was able to open a file over 4GB in size.

answered Oct 18 '10 at 18:56

dhirschl

2,088
13
18

well i work mostly in ubuntu . thanks . but vim itself did a neat job . – Rahul Oct 18 '10 at 19:36
Well say that in your question. – RyanfaeScotland Nov 21 '13 at 12:59
Fully recommended in Windows – luchosrock Jan 28 '14 at 14:56

score 0 · Accepted Answer · answered Oct 18 '10 at 18:28

0

Split them using the gnu (cygwin on Windows) head, tail or cut utilities. Then do work on the file parts.

answered Oct 18 '10 at 18:28

Michael Goldshteyn

71,784
24
131
181

well this file is generated from a tool where i can test db performance . it basically a csv file . is there any way i can split depending upon no of rows i want. – Rahul Oct 18 '10 at 18:42

how to handle large text files?

3 Answers3