FileSystemWatcher Work is Done?

Question

I setup a FsWatcher on a local filesystem directory. I only want to know when files are added to the directory so they can be moved to another filesystem. I seem to be able to detect when the first file is in, but actually I want to know when all files from a given copy operation are done.

If I used Windows Explorer to copy files from one directory to another, Explorer would tell me that there are n seconds left in the transfer, so while there is some activity for the begin-transfer and end-transfer for each file, it appears that there is something for the begin-transfer and end-transfer for all files.

I wonder if there is something similar that I can do just with the .NET Framework. I would like to know when "all" files are in and not just a single file in a "transaction". If there is nothing baked in, maybe I should come up with some kind of waiting/countering in order to only do my activity when a job is "done".

Not sure if I'm making 100% sense on this one, please anyone comment.

Thanks.

Answer 1

The way I have implemented this in the past is to have a 'marker' file with a special name, e.g. 'end'. This file is always the last file written and serves as an indicator for when a multi file transaction is complete. This would work just fine if there is only one thread which is writing these sets of files.

If there are many threads, each of which may be writing a set of files, then you'd need to somehow associate each 'end' file with the set of files it is marking. One way to do this would be to include a GUID (or some other unique identifier) in the filenames, e.g.

{ GUID1.file1.bin, GUID1.file2.bin, GUID1.end }

{ GUID2.file1.bin, GUID2.file2.bin, GUID2.end }

Another way to do it is to zip all the files from each set into a single ZIP file so you only have to watch for single files.

Answer 2

There is no such thing as a transaction for a file copy. You can only watch for single file changes in a directory.

Explorer can do this because it knows what you want to do (copy these x files form here to here).

You do not even know when writing (or more generally modifiying) this file is done. You have to try to exclusively lock the file so you know you are the only one accessing it.

The .end file trick that AdamRalph mentions is the only trick that actually comes close to what you want to do.

hth

Mario

Answer 3

What about writing a file indicating a list of the files to be added as the first step, your application picks it up, reads it and then waits on the files to be written. When the final file is written, you can package them up and send them off. This way if multiple "transactions" are initiated you don't have "end files" stepping on each others toes. Give the indicator file a known pattern name to look for that would be unlikely to occur in regular files and provide some means that multiple indicator files could occur at similar times.

If I were to implement something like this I would probably use some kind of filename like ~<ANOTHER-UNIQUE-GUID>.tmp my file system watcher would be watching for a single file of that format name which in turn tells another file system watcher to wait on the files listed inside. When the files have been moved out, the temporary file can be deleted.

This kind of approach can be threaded easily so that each of your watchers aren't stepping all over each other and transferring/removing files that other watchers are working on.

As suggested previously though, it may be more effective to zip up the files into a single archive and use that, unpackaging them at the destination.