Downloading an XML file via FTP

Question

I have a list of feeds in a database that I use to download a XML file from a FTP server and then parse it. The scrpt is bundled up into a jar file which is run daily using Windows Task Scheduler. Occasionally the request get haulted at grabbing a certain xml file. So far it has happened about 3 times in 2 weeks with no real pattern that I can see.

When it does mess up, I go to the computer it is being run from, I see the command window open and it is stopped before the xml has been fully downloaded. If I close the command window and run the task manually everything will work fine.

The code that I am using to download the xml file is:

private void loadFTPFile(String host, String username, String password, String filename, String localFilename){
        System.out.println(localFilename);
        FTPClient client = new FTPClient();
        FileOutputStream fos = null;

        try {
            client.connect(host);
            client.login(username, password);
            String localFilenameOutput = createFile(assetsPath + localFilename);
            fos = new FileOutputStream(localFilenameOutput);
            client.retrieveFile(filename, fos);

        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                if (fos != null) 
                    fos.close();
                client.disconnect();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }

This function is being called in a loop and when it fails, everything stops and the script doesn't go onto the next feed.

I'm not sure what is happening, possibly the connection being lost, but I would think that the try/catch would catch if that is happening. I'm not sure if a timeout would do the trick or threads need to be used (but I've never worked with threads)

Could anyone point me in the right direction onto why this is happening and what I can do to fix the problem

Answer 1

UPDATE - Set a timeout for the data connection

Since the last file is only partially downloaded, and given the source of FTPClient.retrieveFile(), I think it may be a problem on the server side (something that make it hang, or even die - who knows). Obviously one can't repair the server or even know what's going on there, anyway I suggest to add a timeout with setDataTimeout(int) and catch the possible SocketTimeoutException separately to be logged in a different place and maybe sent to the FTP server admins (along with the time information when it happened) so they can merge the logs and see what's the issue.

OLD ANSWER

I didn't notice that you connect and login for each and every file, so the following is just an optimization not to close the control connection and succesfully logout, but it should not address the problem.

You could start the JVM in debug mode and attach a debugger when it hangs, anyway according to this answer and this thread it can be a timeout problem on the network equipment devices (routers). From the FTPClient Javadoc

During file transfers, the data connection is busy, but the control connection is idle. FTP servers know that the control connection is in use, so won't close it through lack of activity, but it's a lot harder for network routers to know that the control and data connections are associated with each other. Some routers may treat the control connection as idle, and disconnect it if the transfer over the data connection takes longer than the allowable idle time for the router.

One solution to this is to send a safe command (i.e. NOOP) over the control connection to reset the router's idle timer. This is enabled as follows:

ftpClient.setControlKeepAliveTimeout(300); // set timeout to 5 minutes

Answer 2

Do you check the return status of any of the calls or is that the code?

There is a call completePendingCommand() that has to be used on occassion. That may be something to look into.

Also, you won't see an IO exception, I belive it gets repackaged as a CopyStreamException

You might want to also change the return value to a boolean since you trap the exceptions, at least the calling loop will know whether the tranfer happened or not.

private boolean loadFTPFile(String host, String username, String password, String filename, String localFilename){
    System.out.println(localFilename);
    FTPClient client = new FTPClient();
    FileOutputStream fos = null;

    try {
        client.connect(host);

        int reply = client.getReplyCode();

        if (!FTPReply.isPositiveCompletion(reply)){
            client.disconnect();
            System.err.println("FTP server refused connection.");
            return false;
        }


        if (!client.login(username, password)){
            client.logout();
            return false;
        }

        String localFilenameOutput = createFile(assetsPath + localFilename);
        fos = new FileOutputStream(localFilenameOutput);
        boolean result = client.retrieveFile(filename, fos);

        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");

        if (result){
            System.out.println("\tFile Transfer Completed Successfully at: " + sdf.format(Calendar.getInstance().getTime()));

            // ftp.completePendingCommand();
        }
        else {
            System.out.println("\tFile Transfer Failed at: " + sdf.format(Calendar.getInstance().getTime()));
        }

    return result;
    }catch (CopyStreamException cse){
        System.err.println("\n\tFile Transfer Failed at: " + sdf.format(Calendar.getInstance().getTime()));
        System.err.println("Error Occurred Retrieving File from Remote System, aborting...\n");
        cse.printStackTrace(System.err);
        System.err.println("\n\nIOException Stack Trace that Caused the Error:\n");
        cse.getIOException().printStackTrace(System.err);
        return false;
    }catch (Exception e){
        System.err.println("\tFile Transfer Failed at: " + sdf.format(Calendar.getInstance().getTime()));
        System.out.println("Error Occurred Retrieving File from Remote System, aborting...");
        e.printStackTrace(System.err);
        return false;
    } finally {
        try {
            if (fos != null) 
                fos.close();
            client.disconnect();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Answer 3

It's not a threading issue. Chances are it is caused by something in the loop since that code looks like it should clean up just fine. That said, for testing you will probably want to add

catch (Exception e) {
    e.printStackTrace();
}

after the IOException catch clause. It's possible that another exception is being thrown.

Another thing, if you are pulling results from the database result set one at a time and doing the FTP gets, that might be a problem. Unless the results are all brought back by the JDBC call at once, that too could time out. Not all database queries actually return the entire result set to the client at once.