2

I have a poller running on a certain directory every 35s. The files are placed in this directory through a SFTP server. The problem is whenever the polling conflicts with the time when a file is being copied. It picks the incomplete file also which is not yet copied completely.

Can we know the status of a file whether it is in copying mode or copied mode?

vektor
  • 3,312
  • 8
  • 41
  • 71
Arup Mishra
  • 189
  • 2
  • 14
  • 1
    this is generally done by using a temporary flag file for every actual file with `.done` extension or so. If this flag file is missing , file is not fully copied. – Sabir Khan Feb 04 '16 at 07:17

3 Answers3

2

There are several common strategies for file watchers to "know" a file is completely transferred

  1. Poll with time interval, and treat the file to be completely transferred if file size is not changing within an interval. e.g. watch for file existence every 1 minute. Once you see the file exists, monitor its size for every 5 seconds. If file size stays constant for 30 seconds, then treat it as completely transferred.

  2. Have the transfer process create a tagging file after file transfer. e.g. After it completed transferring the file FOO.txt, create an empty FOO.txt.tag. Your file watcher is going to check for existence of FOO.txt.tag and once it exists, you know FOO.txt has been completely transferred

  3. In some special cases that the file is having special format (e.g. a special footer line) then your file watcher can poll the file and see the last lines, and see if they match with the desired pattern

Each method has its pros and cons:

  1. Method 1 affects the transfer process least. Sometimes files are transferred by 3rd party that you have almost no way to tell them to create the tag file as in method 2. However you can tell this method is not 100% reliable, especially under poor network.
  2. Method 2 is the most reliable. However, as said before, there are cases that you have no control on the transfer process
  3. Method 3 is only applicable to special cases

Choose the one that suit your need

Adrian Shum
  • 38,812
  • 10
  • 83
  • 131
1

Have the poller note file sizes. If the size did not change from one round to the next, the file is done downloading.

Can you influence the SFTP server? Can it create a marker file once the download is complete (e.g. '.thisIsAFile.doc.done')?

cypherfox
  • 33
  • 5
  • In some environments SFTP transfers may temporarily slow down due to network lag and thus keeping the file size the same even after the wait interval is over. This happened in my environment. – Jijo Mathew Oct 05 '16 at 06:43
0

Us this for Unix if you are transferring files using winscp or FTP:

public static void isFileReady(File entry) throws Exception {
        long realFileSize = entry.length();
        long currentFileSize = 0;
        do {
            try (FileInputStream fis = new FileInputStream(entry);) {
                currentFileSize = 0;
                while (fis.available() > 0) {
                    byte[] b = new byte[1024];
                    int nResult = fis.read(b);
                    currentFileSize += nResult;
                    if (nResult == -1)
                        break;
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
            System.out.println("currentFileSize=" + currentFileSize + ", realFileSize=" + realFileSize);
        } while (currentFileSize != realFileSize);
    }
Vamsi
  • 619
  • 3
  • 9
  • 22
  • I haven't tested this with SFTP but I did check it against a file partially downloaded with wget, for which it does not work. – Aro May 05 '22 at 15:07