I am trying to implement an application with c# to retrieve only the new added lines in a very large text file (more than 1.5 GB). the new lines added by another service only adds records to this file, I highly need a method to read the last added lines efficiently.
Asked
Active
Viewed 173 times
0
-
How many lines are you looking at reading? The very last line or an arbitrary number of lines? – Fabulous Sep 21 '18 at 16:16
-
@Fabulous, if no way to know the new added lines and read them only, I may read the last 5 records and then compare them with the previous read. – Abdelrahman Sep 21 '18 at 16:23
-
You might want to look at [this question](https://stackoverflow.com/questions/452902/how-to-read-a-text-file-reversely-with-iterator-in-c-sharp) for an efficient technique to read the file in reverse and then decide how many lines you want to read. In it, a technique is offered to read the file backwards so you can leverage that to get the last lines in the file until you have the number you need. – Fabulous Sep 21 '18 at 16:32
-
You could read and store the file stream length before the modification.Then, after each append, position the stream in the last stored position and read to the end of the file. This of course is considering that your file only get appended. – Magnetron Sep 21 '18 at 16:34
-
@Magnetron, is this technique will load all the file bytes in the memory? – Abdelrahman Sep 21 '18 at 16:49
-
@Abdelrahman No, streams don't load all the file in memory, that's what make it ideal for large files. – Magnetron Sep 21 '18 at 16:51
-
This answer is the closest one of an implementation of this logic: https://stackoverflow.com/a/42497394/5762332 just set `var lastReadLength = initialFileSize;` instead of `var lastReadLength = initialFileSize - 1024;` – Magnetron Sep 21 '18 at 16:59
-
@Magnetron Thanks, I really liked your solution – Abdelrahman Sep 21 '18 at 17:01