How to verify file is finished being written before attempting to process it?

Question

Hard to give a good title to my problem but it is as follows. First I'm doing this on windows and it could possibly be used on a linux box also so I'd need the fix to work on both systems. I am monitoring a directory for new files. I basically looking at the directory's files and comparing them over and over and only processing the new files. Problem is I keep getting an error where the file isn't finished being written before I attempt to process.

public class LiveDetectionsProvider extends DetectionsProvider {
protected LiveDetectionsProvider.MonitorDirectory monitorDirectory = null;
protected TimeModel timeModel = null;
private ArrayList<String> loaded = new ArrayList();
private File topLayerFolder = null;
public LiveDetectionsProvider(String directory, String id) {
    super(directory, id);
    timeModel = super.timeModel;
}
/**
 * Initialize the data provider.
 */
public void initialize() {
    try {
        topLayerFolder = new File(directory);
        File[] dir = topLayerFolder.listFiles();
        for (File file : dir) {
            loaded.add(file.getName());
        }
        monitorDirectory = new MonitorDirectory();
        monitorDirectory.execute();
    }
    catch (Exception ex) {
        Logger.getLogger(LiveDetectionsProvider.class.getName()).log(Level.SEVERE, "Failed to read detection\n{0}", ex.getMessage());
    }
    super.initialize();
}
/**
 * Un-initialize the data provider.
 */
public void uninitialize() {
    super.uninitialize();
    if (monitorDirectory != null) {
        monitorDirectory.continuing = false;
    }
}
/**
 * The class that is used to load the detection points in a background
 * thread.
 */
protected class MonitorDirectory extends SwingWorker<Void, Void> {
    public boolean continuing = true;
    /**
     * The executor service thread pool.
     */
    private ExecutorService executor = null;
    /**
     * The completion service that reports the completed threads.
     */
    private CompletionService<Object> completionService = null;
    @Override
    protected Void doInBackground() throws Exception {
        int count = 0;
        executor = Executors.newFixedThreadPool(1);
        completionService = new ExecutorCompletionService<>(executor);
        while (continuing && topLayerFolder != null) {
            File[] dir = topLayerFolder.listFiles();
            Thread.sleep(10);
            ArrayList<File> filesToLoad = new ArrayList();
            for (File file : dir) {
                if (!loaded.contains(file.getName())) {
                    long filesize = 0;
                    boolean cont = true;
                    while (cont) {
                        if (file.length() == filesize) {
                            cont = false;
                            Thread.sleep(3);
                            filesToLoad.add(file);
                        }
                        else {
                            filesize = file.length();
                            Thread.sleep(3);
                        }
                    }
                    Thread.sleep(3);
                }
            }
            for (File file : filesToLoad) {
                timeModel.setLoadingData(LiveDetectionsProvider.this.hashCode(), true);
                completionService.submit(Executors.callable(new ReadDetection(file, false)));
                while (completionService.take() == null) {
                    Thread.sleep(2);
                }
                loaded.add(file.getName());
                count++;
                Logger.getLogger(LiveDetectionsProvider.class.getName()).log(Level.SEVERE, "Detection Message Count:" + count);
            }
            detectionsModel.fireStateChanged(DetectionsModel.CHANGE_EVENT_DETECTIONS);
            timeModel.setLoadingData(LiveDetectionsProvider.this.hashCode(), false);
        }
        return null;
    }
}
}

The file is processed at the line with

completionService.submit(Executors.callable(new ReadDetection(file, false)));

The file at this point still hasnt finished being written and thus fails. I've tried sleeping my thread to slow it down, and I've tried verifying the file size hasn't changed. My test case for this is I'm unzipping a tar file which contains tons of 1,000 KB files.

Please don't forget to add a '?' to questions! Some people do a search in the page for '?' and if none exists in the 'question' go directly to the next (actual) question in line. — Andrew Thompson, Dec 13 '13 at 16:07
Sure. Look at the part of the page between the question tags and your signature! There should be several links, one of which is 'edit'. — Andrew Thompson, Dec 13 '13 at 16:25

score 3 · Answer 1 · answered Dec 13 '13 at 16:13

3

Usually I solve this issue by create a temporary file while the file is being written. Once finish I rename the file and only the renamed file can be process.

answered Dec 13 '13 at 16:13

TheEwook

11,037
6
36
55

Actually this is not as simple as it sounds on Windows. My solution implemented along those lines works in 99.5% cases (more details in http://stackoverflow.com/questions/29923008/how-to-create-then-automically-rename-file-in-java-on-windows) – Piotr Findeisen Apr 28 '15 at 15:18

score 0 · Answer 2 · answered Dec 16 '13 at 17:22

0

Use a "flag file": once file.txt is "finished", indicate this by creating a file.flg - consuming process should wait for .flg to appear.

answered Dec 16 '13 at 17:22

Brian

6,391
3
33
49

score -1 · Accepted Answer · answered Dec 17 '13 at 21:07

First yes it compiles look a the solution I posted in relation to the code In my question. You substitute

For(File file: dir){
while(!file.renameTo(file)){
Thread.sleep(1)
}
// In my code I check to see if the file name is already in the list which 
// contains files that have been previously loaded if its not I add it to a list
// of files to be processed
}

in for

        for (File file : dir) {
            if (!loaded.contains(file.getName())) {
                long filesize = 0;
                boolean cont = true;
                while (cont) {
                    if (file.length() == filesize) {
                        cont = false;
                        Thread.sleep(3);
                        filesToLoad.add(file);
                    }
                    else {
                        filesize = file.length();
                        Thread.sleep(3);
                    }
                }
                Thread.sleep(3);
            }
        }

sorry I forgot to put comment tags // in on the line that said do what every you need to do here.

What it does is it looks at each file in the directory and checks to see if you can rename it if the rename fails it sleeps and continues checking till it sucessfully is able to rename at which point you can do what you need to with the file which in my case was everything after the for loop that was replaced. I'm curious why my awnser was viewed as sub par deleted and locked. This solution does work and solved my problem and would anyone else who's having the same issue attempting to process a file that still being written or copied to a directory thats being monitored for changes.

(I didn't vote.) Doesn't work for me, Windows 10, Java 1.8.0_144. I have one JVM create and write to files. The other tries to detect when the files can be moved with Files.move(). I keep getting moved files of size 0 that are then later still being written to as if the location doesn't matter at all. Anyway, I just added your file.renameTo(file) directly before my move command, and if it would return false, I immediately System.exit(). But the program never exits in that place. — Dreamspace President, Nov 13 '17 at 10:36

How to verify file is finished being written before attempting to process it?

3 Answers3

Linked