It's common knowledge that disk I\O is expensive. I typically read one line at a time using C# from a .txt file. It just crossed my mind that if C# were to give you a way of reading 100 lines a time, it would require less round trips to the disk, and hence decrease time. Is this possible? Does anybody have any ideas on how to make this more efficient. I am going through a file of 1 million rows in about an hour, reading and processing one line at a time. I am wanting to see if I can significantly reduce this time.
Asked
Active
Viewed 77 times
0
-
Possible duplicate: https://stackoverflow.com/q/8037070/495455 – Jeremy Thompson Jun 26 '20 at 01:39
1 Answers
5
A FileStream does this for you, It has an internal buffer and reads the file in chunks. It just so happens File.ReadLines uses a StreamReader which is based on a FileStream and returns an IEnumerable based on lines.. It in essense does everything for you.
My suggestion would be to just use File.ReadLines
If you need more fine-grained control over the internal buffer and chunk size, use FileStream with StreamReader which allows configuration of these concerns, among other things...
TheGeneral
- 79,002
- 9
- 103
- 141
-
that is a very interesting idea. i will try it out one of these days. the drawback i see right now is that you will be reading one big long file in just one statement, and that might take time, and it might be hard to give a progress bar on that. But maybe it won't take that long to read a million rows. I don't know. I have to try it out... – Sir Fartsalot Jun 26 '20 at 01:43
-
3"*one big long file in just one statement*" no its actually not correct, you will be enumerating the file line by line, which will inturn be reading in chunks as you are enumerating it. The file does not get read in one big chunk unless its fits in the standard chunk size of 4k – TheGeneral Jun 26 '20 at 01:44