109

I have an application that writes information to file. This information is used post-execution to determine pass/failure/correctness of the application. I'd like to be able to read the file as it is being written so that I can do these pass/failure/correctness checks in real time.

I assume it is possible to do this, but what are the gotcha's involved when using Java? If the reading catches up to the writing, will it just wait for more writes up until the file is closed, or will the read throw an exception at this point? If the latter, what do I do then?

My intuition is currently pushing me towards BufferedStreams. Is this the way to go?

Joris Schellekens
  • 8,483
  • 2
  • 23
  • 54
Anthony Cramp
  • 4,495
  • 6
  • 26
  • 27
  • 1
    hey, as i'm facing a similar scenario i was wording if you have found a better solution than the accepted one? – Asaf David May 11 '10 at 19:45
  • I know this is an old question, but for the sake of future readers, can you expand on your use case a bit more? Without having more information, one wonders if you're perhaps solving the wrong problem. – user359996 Nov 17 '10 at 20:49
  • 3
    Use a database. These 'read a file while its being written' scenarios end in tears. – user207421 Jan 11 '13 at 23:49
  • @EJP - which DB do you recommend ? I'm guessing MySQL is a good start ? – Caffeinated Sep 24 '15 at 17:41
  • Look into using the [Tailer](https://commons.apache.org/proper/commons-io/javadocs/api-2.4/org/apache/commons/io/input/Tailer.html) from Apache Commons IO. It handles most of the edge cases. – Joshua May 29 '12 at 16:28

9 Answers9

46

Could not get the example to work using FileChannel.read(ByteBuffer) because it isn't a blocking read. Did however get the code below to work:

boolean running = true;
BufferedInputStream reader = new BufferedInputStream(new FileInputStream( "out.txt" ) );

public void run() {
    while( running ) {
        if( reader.available() > 0 ) {
            System.out.print( (char)reader.read() );
        }
        else {
            try {
                sleep( 500 );
            }
            catch( InterruptedException ex ) {
                running = false;
            }
        }
    }
}

Of course the same thing would work as a timer instead of a thread, but I leave that up to the programmer. I'm still looking for a better way, but this works for me for now.

Oh, and I'll caveat this with: I'm using 1.4.2. Yes I know I'm in the stone ages still.

manman
  • 4,743
  • 3
  • 30
  • 42
Joseph Gordon
  • 2,356
  • 2
  • 23
  • 27
  • 1
    Thanks for adding this ... something I never got around to doing. I think Blade's answer of locking the file is also a good one. However, it requires Java 6 (I think). – Anthony Cramp Oct 01 '08 at 23:35
  • @JosephGordon - You'll have to move to the Drone ages one of these days ;-) – TungstenX Nov 30 '16 at 09:36
15

If you want to read a file while it is being written and only read the new content then following will help you achieve the same.

To run this program you will launch it from command prompt/terminal window and pass the file name to read. It will read the file unless you kill the program.

java FileReader c:\myfile.txt

As you type a line of text save it from notepad and you will see the text printed in the console.

public class FileReader {

    public static void main(String args[]) throws Exception {
        if(args.length>0){
            File file = new File(args[0]);
            System.out.println(file.getAbsolutePath());
            if(file.exists() && file.canRead()){
                long fileLength = file.length();
                readFile(file,0L);
                while(true){

                    if(fileLength<file.length()){
                        readFile(file,fileLength);
                        fileLength=file.length();
                    }
                }
            }
        }else{
            System.out.println("no file to read");
        }
    }

    public static void readFile(File file,Long fileLength) throws IOException {
        String line = null;

        BufferedReader in = new BufferedReader(new java.io.FileReader(file));
        in.skip(fileLength);
        while((line = in.readLine()) != null)
        {
            System.out.println(line);
        }
        in.close();
    }
}
tiger.spring
  • 492
  • 3
  • 3
  • 3
    Isn't there the possibility that, between the time the file is read and the time the file length is taken, the external process could add more data to the file. If so, this would result in the reading process missing data written to the file. – JohnC Jul 18 '16 at 19:08
  • 2
    This code consumes a lot of CPU because the loop does not have a thread.sleep call in it. Without adding a small amount of delay this code tends to keep the CPU very busy. – ChaitanyaBhatt May 11 '17 at 22:31
  • 1
    Both comments above are true and additionally this `BufferedReader` creation in each loop call is so pointless. With it being created once skipping would be not necessary, as buffered reader would read new lines as they arrive. – Piotr Jun 19 '20 at 18:17
8

You might also take a look at java channel for locking a part of a file.

http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html

This function of the FileChannel might be a start

lock(long position, long size, boolean shared) 

An invocation of this method will block until the region can be locked

Frederic Morin
  • 8,733
  • 4
  • 28
  • 27
7

I totally agree with Joshua's response, Tailer is fit for the job in this situation. Here is an example :

It writes a line every 150 ms in a file, while reading this very same file every 2500 ms

public class TailerTest
{
    public static void main(String[] args)
    {
        File f = new File("/tmp/test.txt");
        MyListener listener = new MyListener();
        Tailer.create(f, listener, 2500);

        try
        {
            FileOutputStream fos = new FileOutputStream(f);
            int i = 0;
            while (i < 200)
            {
                fos.write(("test" + ++i + "\n").getBytes());
                Thread.sleep(150);
            }
            fos.close();
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
    }

    private static class MyListener extends TailerListenerAdapter
    {
        @Override
        public void handle(String line)
        {
            System.out.println(line);
        }
    }
}
Community
  • 1
  • 1
ToYonos
  • 16,469
  • 2
  • 54
  • 70
3

The answer seems to be "no" ... and "yes". There seems to be no real way to know if a file is open for writing by another application. So, reading from such a file will just progress until content is exhausted. I took Mike's advice and wrote some test code:

Writer.java writes a string to file and then waits for the user to hit enter before writing another line to file. The idea being that it could be started up, then a reader can be started to see how it copes with the "partial" file. The reader I wrote is in Reader.java.

Writer.java

public class Writer extends Object
{
    Writer () {

    }

    public static String[] strings = 
        {
            "Hello World", 
            "Goodbye World"
        };

    public static void main(String[] args) 
        throws java.io.IOException {

        java.io.PrintWriter pw =
            new java.io.PrintWriter(new java.io.FileOutputStream("out.txt"), true);

        for(String s : strings) {
            pw.println(s);
            System.in.read();
        }

        pw.close();
    }
}

Reader.java

public class Reader extends Object
{
    Reader () {

    }

    public static void main(String[] args) 
        throws Exception {

        java.io.FileInputStream in = new java.io.FileInputStream("out.txt");

        java.nio.channels.FileChannel fc = in.getChannel();
        java.nio.ByteBuffer bb = java.nio.ByteBuffer.allocate(10);

        while(fc.read(bb) >= 0) {
            bb.flip();
            while(bb.hasRemaining()) {
                System.out.println((char)bb.get());
            }
            bb.clear();
        }

        System.exit(0);
    }
}

No guarantees that this code is best practice.

This leaves the option suggested by Mike of periodically checking if there is new data to be read from the file. This then requires user intervention to close the file reader when it is determined that the reading is completed. Or, the reader needs to be made aware the content of the file and be able to determine and end of write condition. If the content were XML, the end of document could be used to signal this.

Anthony Cramp
  • 4,495
  • 6
  • 26
  • 27
2

There are a Open Source Java Graphic Tail that does this.

https://stackoverflow.com/a/559146/1255493

public void run() {
    try {
        while (_running) {
            Thread.sleep(_updateInterval);
            long len = _file.length();
            if (len < _filePointer) {
                // Log must have been jibbled or deleted.
                this.appendMessage("Log file was reset. Restarting logging from start of file.");
                _filePointer = len;
            }
            else if (len > _filePointer) {
                // File must have had something added to it!
                RandomAccessFile raf = new RandomAccessFile(_file, "r");
                raf.seek(_filePointer);
                String line = null;
                while ((line = raf.readLine()) != null) {
                    this.appendLine(line);
                }
                _filePointer = raf.getFilePointer();
                raf.close();
            }
        }
    }
    catch (Exception e) {
        this.appendMessage("Fatal error reading log file, log tailing has stopped.");
    }
    // dispose();
}
Community
  • 1
  • 1
Rodrigo Menezes
  • 245
  • 7
  • 20
2

You can't read a file which is opened from another process using FileInputStream, FileReader or RandomAccessFile.

But using FileChannel directly will work:

private static byte[] readSharedFile(File file) throws IOException {
    byte buffer[] = new byte[(int) file.length()];
    final FileChannel fc = FileChannel.open(file.toPath(), EnumSet.of(StandardOpenOption.READ));
    final ByteBuffer dst = ByteBuffer.wrap(buffer);
    fc.read(dst);
    fc.close();
    return buffer;
}
bebbo
  • 2,830
  • 1
  • 32
  • 37
1

Not Java per-se, but you may run into issues where you have written something to a file, but it hasn't been actually written yet - it might be in a cache somewhere, and reading from the same file may not actually give you the new information.

Short version - use flush() or whatever the relevant system call is to ensure that your data is actually written to the file.

Note I am not talking about the OS level disk cache - if your data gets into here, it should appear in a read() after this point. It may be that the language itself caches writes, waiting until a buffer fills up or file is flushed/closed.

Matthew Schinckel
  • 35,041
  • 6
  • 86
  • 121
0

I've never tried it, but you should write a test case to see if reading from a stream after you have hit the end will work, regardless of if there is more data written to the file.

Is there a reason you can't use a piped input/output stream? Is the data being written and read from the same application (if so, you have the data, why do you need to read from the file)?

Otherwise, maybe read till end of file, then monitor for changes and seek to where you left off and continue... though watch out for race conditions.

Mike Stone
  • 44,224
  • 30
  • 113
  • 140