Reading new data from a constantly updating file in Java

Question

I have a log file which is constantly updated with new lines of data. I need to get new added data in java as soon as it's written. For now my solution is:

public static void readNonStop(String filename, boolean goToEnd, FileReadCallback readCallback) {
    if(readCallback == null) {
        return;
    }
    try {
        BufferedReader br = new BufferedReader(new FileReader(filename));
        try {
            String line = br.readLine();
            int lineNumber = 0;

            if(goToEnd) {
                while(br.readLine() != null) {}
            }

            while (true) {
                if(line != null) {
                    readCallback.onRead(lineNumber++, line);
                } else {
                    Thread.sleep(1);
                }
                line = br.readLine();
            }
        } finally {
            br.close();
        }
    } catch (Exception e) {
        e.printStackTrace();
    }
}

But I have a feeling, that there should be a better way. I don't like the idea of a constant runnin loop with a "sleep" inside and would prefer some sort of an event driven approach.

If I rely on FileSystem events to re-open the file each time it is modified, it itroduces a delay.

What is the correct way of doing it for this situation?

Thanks in advance!

As it stands, you do not have a concrete question. As such your post would be better suited over at [Code Review](https://codereview.stackexchange.com/) — hotzst, Dec 27 '18 at 18:49
Have you looked at Java Watch Service. Its way better than trying to implement your own logic. Take a look at this posting - https://stackoverflow.com/questions/16251273/can-i-watch-for-single-file-change-with-watchservice-not-the-whole-directory — Dinesh, Dec 27 '18 at 19:00
I'm currently using a WatchService, but the problem is, that this introduces a delay because you need to open and close the file all the time. The data is needed as soon as possible. — Arthur, Dec 27 '18 at 19:26
Have you tried the apache commons TailerListener? Javadoc: https://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/input/Tailer.html Implementation: https://commons.apache.org/proper/commons-io/apidocs/src-html/org/apache/commons/io/input/Tailer.html#line.120 — JuanMoreno, Dec 27 '18 at 19:55

score 2 · Accepted Answer · answered Dec 27 '18 at 19:49

2

Files are not designed to be a messaging solution. Even using TCP over loopback can have a delay of 10 - 30 microseconds. Without changing the file format, your solution is likely to be the fastest.

NOTE: you don't have to sleep for a whole millisecond. You can use Thread.yield() or LockSupport.parkNanos(100_000); For a more complex strategy, you can have a class like LongPauser which backs off in a configurable way.

BTW I implemented a solution to write/read files in a low latency way called Chronicle Queue. This has sub-microsecond latencies using a binary format for speed.

NOTE: You can just to the end by skipping all the bytes available() when you open the file as a FileInputStream. This might result in an incomplete line depending on how your buffering works.

answered Dec 27 '18 at 19:49

Peter Lawrey

525,659
79
751
1,130

Thanks for your answer. Currently the only thing I don't like about this solution is the strain on the CPU if there is no "Thread.sleep". With at least 1 millisecond of sleep the CPU usage drops from 40% to 0-1%. Using a WatchService introduces a delay of up to 5-6 seconds in some cases which is unnacceptable. The thing is, this is the only way to get needed information, so I have no way to switch communication methods. I should use FileInputStream instead though. – Arthur Dec 27 '18 at 19:59
You might find sleeping for longer than 10 ms is fine. What LongPauser does is sleep for longer and longer periods of time until you call reset(). This minimises a delay which things are busy. – Peter Lawrey Dec 27 '18 at 20:08
1

At Code Review I got a suggestion to use the [Tailer](http://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/input/Tailer.html) class from Apache commons. Looking at it's code, looks like I had the right idea. While a bit more robust, `Tailer` uses the same logic as my code, it reads in a loop with a `Thread.sleep` delay. – Arthur Dec 27 '18 at 20:45
*Files are not designed to be a messaging solution.* Indeed. This is a very bad design - especially if Java is the tool used. The reading of a changing/growing file is hard enough to make reliable using low-level C code. With Java it's *de facto* impossible because you can't control if the JVM caches the file size, for example. – Andrew Henle Dec 28 '18 at 08:26
@AndrewHenle, I am writing an RGB keyboard integration for a game and the only way to get information out of it (without obviously hacking into the memory) is by reading it's log files which are instantly updated as soon as an event happens in the game. – Arthur Dec 28 '18 at 09:58

Reading new data from a constantly updating file in Java

1 Answers1