26

Currently I'm working on the project that does processing files from source directory in one of it's routines. There's a Java process that's looking for specified directory and tries to read and process files if they exist. Files are quit large and updates by other thirdparty process. The question is how can I check if the file is completely written? I'm trying to use file.length() but looks like even if writing process hasn't been completed it returns actual size. I have a feeling that solution should be platform dependent. Any help would be appreciated.

UPDATE: This question is not really different from the duplicate but it has an answer with working code snippet that is highly rated.

Viktor Stolbin
  • 2,899
  • 4
  • 32
  • 53
  • 5
    Have you considered using file locks? – cklab Jun 28 '12 at 07:01
  • 1
    Thanks. Checking this. Can we be sure locks are always created by writing process? – Viktor Stolbin Jun 28 '12 at 07:07
  • 2
    Well, the notion is if you attempt to obtain a lock to it, you should get an exception. I just came across this as well: http://docs.oracle.com/javase/7/docs/api/java/io/File.html#canWrite%28%29 Never used it but you can give it a shot to see how it works out. Looks interesting. – cklab Jun 28 '12 at 07:17
  • 2
    I don't know why that question is marked as duplicate: the answer in the other question is when you have control of the writing process. – Matthieu May 07 '16 at 13:22

6 Answers6

22

I got the solution working:

private boolean isCompletelyWritten(File file) {
    RandomAccessFile stream = null;
    try {
        stream = new RandomAccessFile(file, "rw");
        return true;
    } catch (Exception e) {
        log.info("Skipping file " + file.getName() + " for this iteration due it's not completely written");
    } finally {
        if (stream != null) {
            try {
                stream.close();
            } catch (IOException e) {
                log.error("Exception during closing file " + file.getName());
            }
        }
    }
    return false;
}

Thanks to @cklab and @Will and all others who suggested to look in "exclusive lock" way. I just posted code here to make other interested in people use it. I believe the solution with renaming suggested by @tigran also works but pure Java solution is preferable for me.

P.S. Initially I used FileOutputStream instead of RandomAccessFile but it locks file being written.

Doug Porter
  • 7,721
  • 4
  • 40
  • 55
Viktor Stolbin
  • 2,899
  • 4
  • 32
  • 53
  • 4
    This works on Winows but not on Linux. – Nilesh Mar 18 '15 at 05:25
  • @Nelish this is weird because I initially implemented it on Linux for Linux – Viktor Stolbin Mar 21 '15 at 21:59
  • Well I tried it on UNIX. A winscp or ftp was transferring file while I tried to lock and I was able to! – Nilesh Mar 22 '15 at 04:30
  • @Nilesh this might be a case if transferring process doesn't acquire exclusive lock, I don't know details. It worked with rsync for me. – Viktor Stolbin Mar 22 '15 at 17:10
  • rsync is using temporary file and renames it to final once finished, that's why it worked. – rkosegi Jul 26 '15 at 15:46
  • 2
    I've tried this solution in Windows and works fine, but if the source file is in "Read Only" mode, it will never process it. Is there a solution for that case? – Daniel Rodríguez Apr 20 '16 at 11:20
  • 2
    Solution should be platform agnostic. – Avik Aug 01 '16 at 09:56
  • This solution works on some versions of windows/java, but newer versions seem to allow reading and writing to a file even if it's being written by other process (file.canWrite() will be true while a file is being copied). In this case I used the file.renameTo(file) to check if it's still being written. – ccallendar May 10 '17 at 18:29
11

Does the producer process close the file when its finished writing? If so, trying to open the file in the consumer process with an exclusive lock will fail if the producer process is still producing.

Will
  • 73,905
  • 40
  • 169
  • 246
6

One simple solution I've used in the past for this scenario with Windows is to use boolean File.renameTo(File) and attempt to move the original file to a separate staging folder:

boolean success = potentiallyIncompleteFile.renameTo(stagingAreaFile);

If success is false, then the potentiallyIncompleteFile is still being written to.

JoshDM
  • 4,939
  • 7
  • 43
  • 72
  • 1
    Good suggestion for Windows users, but it's not platform-agnostic. On Linux/Unix you can delete and rename files that are still open by other processes. – Erwin Bolwidt Feb 16 '17 at 03:34
  • 2
    That would be why my answer says "with Windows" and not "for all systems". – JoshDM Feb 19 '17 at 15:51
3

I don't think there is a general solution for that. Looking for a file size is wrong as some applications can set file size prior any write call. One of the possibility is to use locking. This will require that writer accrue a write lock ( or exclusive lock ). If you can't modify writer, then you may use tools provided by OS, like fuser on Linux to see is there a process which still accesses the file.

kofemann
  • 4,217
  • 1
  • 34
  • 39
2

If you plan to use this code on a single platform, you may be able to use NIO's FileLock facility. But read the documentation carefully, and note that on many platforms, the lock is only advisory.

Another approach is to have one process write the file with a name that your process won't recognize, then rename the file to a recognizable name when the write is complete. On most platforms, the rename operation is atomic if the source and destination are the same file system volume.

JAVAGeek
  • 2,674
  • 8
  • 32
  • 52
  • Good point. Thanks. Will do it if no success with java. It's a bit hard to change the producer behavior due some circumstances. – Viktor Stolbin Jun 28 '12 at 07:13
1

If you can use Java 1.7, take a look at the NIO tools, specifically java.nio.channels.FileChannel

here is an example of locking the file and reading it.

Community
  • 1
  • 1
sevensevens
  • 1,703
  • 16
  • 27