How can I read big txt file?

Question

i have a 1000 text file and i want read single to single and Each file has a 4700000 record,for example one of line in file is:

43266200 6819           43295200 1393/05/23 14:28:45    113      1

and i want save into sql server for example:

field1:43266200
field2:6819

how can i do this?

How big is each line (record)? Is the record fixed length? I'd recommend a `StreamReader` to avoid possible out of memory issues. — Tim, Nov 02 '14 at 07:39
You will probably need [C# Reading a File Line By Line](http://stackoverflow.com/questions/1271225/c-sharp-reading-a-file-line-by-line) if your file is big enough and don't want to load it in memory all at once. — mihai, Nov 02 '14 at 07:50
So you need a book telling you how to read a text file? Tried the documentation? — TomTom, Nov 02 '14 at 08:21

score 2 · Answer 1 · answered Nov 02 '14 at 07:45

2

var seperators = " ".ToCharArray();
foreach(var line in File.ReadLines(path))
{
  var fields = line.Split(seperators, StringSplitOptions.RemoveEmptyEntries);
  //now you have fields[0] and fields[1], save them in your database
}

answered Nov 02 '14 at 07:45

Hamid Pourjam

20,441
9
58
74

ReadLines isn't reliable for such big files (OutOfMemoryExceptions and so). A line to line streamreader should perform better. – AFract Nov 02 '14 at 08:40
1

`ReadLines` returns an `IEnumerable` and it is designed to be used with large files because it just fetch current line in memory not the whole file, maybe you are talking about `ReadAllLines` which returns string[] and read the whole file at once. – Hamid Pourjam Nov 02 '14 at 10:05
You're right, sorry. I was actually thinking to ReadAllLines. – AFract Nov 02 '14 at 10:50

score 1 · Answer 2 · answered Nov 02 '14 at 07:38

1

This may help you

var message ="43266200 6819           43295200 1393/05/23 14:28:45    113      1";
//Split your data into pieces
var messages=message.Split(' ').Where( o => !string.IsNullOrEmpty(o));
var i=0;
foreach(var item in messages)
{
  // do whatever you wanna to do with pieces
   Console.Write( "field {0}:{1}",++i,item);
}

answered Nov 02 '14 at 07:38

Anik Islam Abhi

25,137
8
58
80

4

This will result in a bunch of blank fields, since you're splitting on a single space. At the very lest use `StringSplitOptions.RemoveEmpties`. – Tim Nov 02 '14 at 07:40

furkle · Answer 3 · 2014-11-02T07:54:24.447

If you're reading the text from a file, and you can reasonably assume that the space character will be your only delimiter, you should use the String.Split() method to tokenize each line:

// instantiate FileInfo of your file as yourFile

foreach (string line in yourFile.ReadLines())
{
    string[] lineTokens = line.Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
}

String.Split() allows you to separate any string into a string[] of substrings based on the char delimiters you provide in the first argument. The second argument in the code above is one of the values in the StringSplitOptions enumeration, which has values of either None (provide all strings) or RemoveEmptyEntries (do not return any substrings that consist solely of delimiter characters).

Then, from there, you can iterate through lineTokens and assemble an object from each token, or you can assemble an SQL query where any given index corresponds to a column in the row you intend to add.

How can I read big txt file?

3 Answers3