Java subscribing to read a new lines from a big file

Question

I would like to read syslog messages from my java application. I will be looking into file /var/log/messages. That file can be huge. I would like to have a thread:

1. Which will constantly check if some new data has appeared in that file
2. Will execute specific actions if specific lines has been found

What is the best(efficient) way to realize step 1. How can I constantly read just new lines from the file?

Answer 1

Launch a Thread and Get your file's instance for the first time that you are trying to check the changes for. This would be your starting point. Now Use the Java Diff Utility to check if your file got any new lines appended or any sort of changes that took place in your file.

Let's call this instance of the file currentFile

You can use this utility to keep checking for new data.

https://code.google.com/p/java-diff-utils/

Diff Utils library is an OpenSource library for performing the comparison operations between texts: computing diffs, applying patches, generating unified diffs or parsing them, generating diff output for easy future displaying (like side-by-side view) and so on.

Here is a Sample code to check for changes

import difflib.*;

public class BasicJavaApp_Task1 {
        // Helper method for get the file content
        private static List<String> fileToLines(String filename) {
                List<String> lines = new LinkedList<String>();
                String line = "";
                try {
                        BufferedReader in = new BufferedReader(new FileReader(filename));
                        while ((line = in.readLine()) != null) {
                                lines.add(line);
                        }
                } catch (IOException e) {
                        e.printStackTrace();
                }
                return lines;
        }

        public static void main(String[] args) {
                List<String> original = fileToLines("originalFile.txt");
                List<String> revised  = fileToLines("revisedFile.txt");

                // Compute diff. Get the Patch object. Patch is the container for computed deltas.
                Patch patch = DiffUtils.diff(original, revised);

                for (Delta delta: patch.getDeltas()) {
                        System.out.println(delta);
                }
        }
}

You can use the Delta to check for your particular data that you are looking for.

Now it is up to you, to decide what is the frequency that you want the comparisons to happen with. Like Every 1000ms, or 10000ms and so on.

Also make sure After Every Comparison You should update your currentFile

For Step 2: In order to send Callbacks You can do the following

Use a ScheduleExecutorService

ScheduledExecutorService ses = Executors.newScheduledThreadPool(1);
void registerCallback() {
  ses.schedule(new MyCommand(), 30, TimeUnit.SECONDS);
}

It returns a Future which can be used to cancel the execution if you want to, or to obtain a value returned by MyCommand.

Should you want to schedule a command that will keep repeating on a timely fashion, you could use other scheduling methods: scheduleAtFixedRate and scheduleWithFixedDelay.

If you need to reschedule on some condition only, or with different rates or intervals, one technique I've used is to pass the ScheduledExecutorService to your command (ie, new MyCommand(ses)) and let it rescheduled itself or a new command with the appropriate delay:

class MyCommand implements Runnable {
  private final ScheduledExecutorService ses;
  MyCommand(ScheduledExecutorService ses) { this.ses = ses; }
  private boolean shouldReschedule() { ... }
  private int getRescheduleTimeoutMs() { ... }
  @Override void run() {
    // do work
    ...
    // reschedule if needed
    if (shouldReschedule()) {
      // reschedule this command:
      ses.schedule(this, getRescheduleTimeoutMs(), TimeUnit.MILLISECONDS);
      // or else a new one:
      ses.schedule(new MyCommand(ses), ...);
    }
  }
}