how can I read an updating file?

Question

I am trying to read a log file for streaming.Also that file is appending new records.

while(true){
for (line <- Source.fromFile(filename).getLines()) {
    pub.publish(topic, line)}}

With this above code it again read file from the beginning . How I can overcome this situation.Next time I want it read from the upcoming records not the entire records

Answer 1

This question has been answered here: Java IO implementation of unix/linux "tail -f"

I am not familiar with scala, but it seems that the logic is simple and clear enough to convert into scala syntax. Hope this could help.

Answer 2

BufferedReader to the rescue

I have solved this problem using java.io.BufferedReader.

Short solution

while (true) {

   if (reader.readLine() != null) {
    //keep reading and pushing the data to function f
    f(reader.readLine())
   } else {
    Try {
     Thread.sleep(checkingTime)
    }
   }//end of if

  }//end of while

More Info

Lets say stream.txt is the file which is getting updated continuously by another process.

Now if we want to read progressively from the file without starting from the start.

The trick is BufferReader's readLine method returns null when it reaches EOF, wait when it gives null and start checking if anything is written to the file using readLine and the start reading using same readLine.

Here is the code snippet for progressive reading.

object ProgressiveReader {

 def read(file: File, checkingTime: Long = 1000)(f: String => Unit): Unit = {

  import java.io._

  val reader = new BufferedReader(new InputStreamReader(new FileInputStream(file)))

  while (true) {

   if (reader.readLine() != null) {
    //keep reading and pushing the data to function f
    f(reader.readLine())
   } else {
    Try {
     Thread.sleep(checkingTime)
    }
   }//end of if

  }//end of while

}

 //usage
 def main(args: Array[String]): Unit = {
  read(new File("stream.txt")) { str =>
   println(str)
  }
 }

}