0

I have a large text file consisting of thousands of lines. I keep adding new lines to the text file on a daily basis. These lines are parsed and added into the database. In Java, is there a way I can read only the new lines being added to the text file rather than reading all the lines right from the start every time I run the java application?

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
Rithviik Srinivasan
  • 3
  • 2
  • Please, in order to do a correct question, you should read the [How to ask](https://stackoverflow.com/help/how-to-ask) section and after that edit this question. Thanks – Inazense Sep 24 '20 at 09:06
  • If you know that the file will only ever be appended to, then you can remember the size after the last time and just seek to that position the next time you read, but there's a bit of a race condition since if the file size changes while you read, that might not be accurate. – Joachim Sauer Sep 24 '20 at 09:23
  • this already has answers here: https://stackoverflow.com/questions/8664705/how-to-read-file-from-end-to-start-in-reverse-order-in-java – ahrooran Sep 24 '20 at 11:09
  • Does this answer your question? [How to read file from end to start (in reverse order) in Java?](https://stackoverflow.com/questions/8664705/how-to-read-file-from-end-to-start-in-reverse-order-in-java) – ahrooran Sep 24 '20 at 11:10

1 Answers1

0

You can store the last read line count. And then, open the file as a stream and skip the count that's processed already.

    //offset = already processed line count
    Stream<String> lines = Files.lines(Paths.get(fileLocation));
    lines.skip(offset);
    lines.forEach(line -> {
        // process
        // offset ++;
    });
Niyas
  • 229
  • 1
  • 7
  • 1
    This will still have to read all the "skipped" files. Remembering the number of lines read basically means there's no way to avoid re-reading them. Instead you'll have to remember the byte-offset of the last byte read and use that (but that's can't be done in such a nice, compact way). – Joachim Sauer Sep 24 '20 at 11:04
  • Joachim Sauer - Thanks for this, Can u please share sample code based on this byte offset as i am totally confused and tried many things . Appreciate ur urgent reply Thanks again – Rithviik Srinivasan Sep 24 '20 at 11:15