How to know whether a file copying is 'in progress'/complete in java (1.6)

Question

I am writing a directory monitoring utility in java(1.6) using polling at certain intervals using lastModified long value as the indication of change. I found that when my polling interval is small (seconds) and the copied file is big then the change event is fired before the actual completion of file copying.

I would like to know whether there is a way I can find the status of file like in transit, complete etc.

Environments: Java 1.6; expected to work on windows and linux.

Answer 1

There are two approaches I've used in the past which are platform agnostic.

1/ This was for FTP transfers where I controlled what was put, so it may not be directly relevant.

Basically, whatever is putting a file file.txt will, when it's finished, also put a small (probably zero-byte) dummy file called file.txt.marker (for example).

That way, the monitoring tool just looks for the marker file to appear and, when it does, it knows the real file is complete. It can then process the real file and delete the marker.

2/ An unchanged duration.

Have your monitor program wait until the file is unchanged for N seconds (where N is reasonably guaranteed to be large enough that the file will be finished).

For example, if the file size hasn't changed in 60 seconds, there's a good chance it's finished.

There's a balancing act between not thinking the file is finished just because there's no activity on it, and the wait once it is finished before you can start processing it. This is less of a problem for local copying than FTP.

Answer 2

This solution worked for me:

File ff = new File(fileStr);

if(ff.exists()) {

    for(int timeout = 100; timeout>0; timeout--) {
        RandomAccessFile ran = null;

        try {
            ran = new RandomAccessFile(ff, "rw");
            break; // no errors, done waiting
        } catch (Exception ex) {
            System.out.println("timeout: " + timeout + ": " + ex.getMessage());
        } finally {
            if(ran != null) try {
                ran.close();
            } catch (IOException ex) {
                //do nothing
            }

            ran = null;
        }

        try {
            Thread.sleep(100); // wait a bit then try again
        } catch (InterruptedException ex) {
            //do nothing
        }
    }

    System.out.println("File lockable: " + fileStr +
                 (ff.exists()?" exists":" deleted during process"));
} else {
    System.out.println("File does not exist: " + fileStr);
}

This solution relies on the fact that you can't open the file for writing if another process has it open. It will stay in the loop until the timeout value is reached or the file can be opened. The timeout values will need to be adjusted depending on the application's actual needs. I also tried this method with channels and tryLock(), but it didn't seem to be necessary.

Answer 3

Do you mean that you're waiting for the lastModified time to settle? At best that will be a bit hit-and-miss.

How about trying to open the file with write access (appending rather than truncating the file, of course)? That won't succeed if another process is still trying to write to it. It's a bit ugly, particularly as it's likely to be a case of using exceptions for flow control (ick) but I think it'll work.

Answer 4

We used to monitor the File Size change for determine whether the File is inComplete or not.

we used Spring integration File endpoint to do the polling for a directory for every 200 ms.

Once the file is detected(regardless of whether it is complete or not), We have a customer File filter, which will have a interface method "accept(File file)" to return a flag indicating whether we can process the file.

If the False is returned by the filter, this FILE instance will be ignored and it will be pick up during the next polling for the same filtering process..

The filter does the following:

First, we get its current file size. and we will wait for 200ms(can be less) and check for the size again. If the size differs, we will retry for 5 times. Only when the file size stops growing, the File will be marked as COMPLETED.(i.e. return true).

Sample code used is as the following:

public class InCompleteFileFilter<F> extends AbstractFileListFilter<F> {

    protected Object monitor = new Object();
    @Override
    protected boolean accept(F file) {
    synchronized (monitor){

        File currentFile = (File)file;

        if(!currentFile.getName().contains("Conv1")){return false;}

        long currentSize = currentFile.length();
        try { Thread.sleep(200); } catch (InterruptedException e) { e.printStackTrace(); }
        int retryCount = 0;
        while(retryCount++ < 4 && currentFile.length() > currentSize){
            try { Thread.sleep(200); } catch (InterruptedException e) { e.printStackTrace(); }
        }

        if(retryCount == 5){
            return false;
        }else{
            return true;
        }
    }
    }
}

Answer 5

If I understood the question correctly, you're looking for a way to distinguish whether the copying of a file is complete or still in progress?

How about comparing the size of the source and destination file (i.e. file.length())? If they're equal, then copying is complete. Otherwise, it's still in progress.

I'm not sure it's efficient since it would still require polling. But it "might" work.

Answer 6

You could look into online file upload with progressbar techniques - they use OutputStreamListener and custom writer to notify the listener about bytes written.

http://www.missiondata.com/blog/java/28/file-upload-progress-with-ajax-and-java-and-prototype/

File Upload with Java (with progress bar)