0

I am developing a winform application that moves files between directories in a SAN. The app searches a list of files in directories with over 200,000 files (per directory) in the SAN's Directory, and then moves the found files to another directory.

For example I have this path:

\san\xxx\yyy\zzz

I perform the search in \zzz and then move the files to \yyy, but while I'm moving the files, another app is inserting more files into the xxx, yyy and zzz directories.

I don't want to impact the access or performance to other applications that use these directories.

The file's integrity is also a big concern, because when the application moves the files to \yyy directory another app uses those files.

All of these steps should run in parallel.

What methods could I use to achieve this? How can I handle concurrency?

Thanks in advance.

Renato Reyes
  • 189
  • 1
  • 1
  • 10
  • Related: [Retrieving files from directory that contains large amount of files](http://stackoverflow.com/questions/7865159/retrieving-files-from-directory-that-contains-large-amount-of-files) and if the files are large, [Windows Offloaded Data Transfers](https://technet.microsoft.com/en-us/library/Hh831628.aspx) – Mitch Jul 23 '15 at 00:47

2 Answers2

0

Using the published .net sources, one can see how the move file operation works Here. Now if you follow the logic you'll find there's a call to KERNEL32. The MSDN documentation for this states

To perform this operation as a transacted operation, use the MoveFileTransacted function.

Based on that I'd say the move method is a safe/good way to move a file.

Searching through a large number of files is always going to take time. If you need to increase the speed of this operation, I'd suggest think out of side of the box to achieve this. For instance, I've seen cache buckets produced for both letters and dates. Using some sort of cache system, you could search for files in a quicker manner, say by asking the cache bucket for files starting with "a" for "these files like this starting with a" file, or files between these dates - etc.

Plebsori
  • 1,075
  • 9
  • 23
0

Presumably, these files are used by something. For the purposes of this conversation, let's assume they are indexed in an SQL Database.

TABLE: Files

File ID        Filename
-------------- ----------------------------
1              \\a\b\aaaaa.bin
2              \\a\b\bbbbb.bin

To move them to \\a\c\ without impacting access, you can execute the following:

  1. Copy the file from \\a\b\aaaa.bin to \\a\c\aaaa.bin
  2. Update the database to point to \\a\c\aaaa.bin
  3. Once all open sessions have been closed for \\a\b\aaaa.bin, delete the old file

If the files are on the same volume, you could also use a hardlink instead of a copy.

Mitch
  • 21,223
  • 6
  • 63
  • 86