Java FTP using Apache commons throws "IOException caught while copying"

Question

I have made a JavaFX application which includes uploading large files (> 1GB) to a server. Every time I got the same error on same place. Any Suggestions what am I doing wrong here.

ftpclient.connect(server, port);
ftpclient.login(ftpuser, ftppass);
ftpclient.enterLocalPassiveMode();
ftpclient.setKeepAlive(true);
ftpclient.setControlKeepAliveTimeout(3000);

Task<Void> copyMnt = new Task<Void>() {
@Override
protected Void call(){
  try {                              
      new Thread(new FTPHandler(ftpclient, source , dest)).run();
  } catch (IOException ex) {
     Logger.getLogger(MyAppController.class.getName()).log(Level.SEVERE, null, ex); 
  }
 return null;
  }
};


new Thread(copyMnt).start();

Now on the FTPHandler Class

// The constructor will set the ftpclient, source and destinations.
@Override

public void run() {
    try {
        uploadDirectory(this.getClient(), this.getDest(), this.getSrc(), "");
    } catch (IOException ex) {
        Logger.getLogger(FTPHandler.class.getName()).log(Level.SEVERE, null, ex);
    }
}

public static void uploadDirectory(FTPClient ftpClient,
        String remoteDirPath, String localParentDir, String remoteParentDir)
        throws IOException {

    File localDir = new File(localParentDir);
    File[] subFiles = localDir.listFiles();
    if (subFiles != null && subFiles.length > 0) {
        for (File item : subFiles) {
            String remoteFilePath = remoteDirPath + "/" + remoteParentDir
                    + "/" + item.getName();
            if (remoteParentDir.equals("")) {
                remoteFilePath = remoteDirPath + "/" + item.getName();
            }

            if (item.isFile()) {
                // upload the file
                String localFilePath = item.getAbsolutePath();
                java.util.Date date= new java.util.Date();

                System.out.println(new Timestamp(date.getTime()) + "  : Uploading :: " + localFilePath + " to " + remoteFilePath);
                boolean uploaded = uploadSingleFile(ftpClient,
                        localFilePath, remoteFilePath);
                if (uploaded) {
                    System.out.println("Success : "
                            + remoteFilePath);
                } else {
                    System.out.println("Failed : "
                            + localFilePath);
                }
            } else {
                // create directory on the server
                boolean created = ftpClient.makeDirectory(remoteFilePath);
                if (created) {
                    System.out.println("CREATED the directory: "
                            + remoteFilePath);
                } else {
                    System.out.println("COULD NOT create the directory: "
                            + remoteFilePath);
                }

                // upload the sub directory
                String parent = remoteParentDir + "/" + item.getName();
                if (remoteParentDir.equals("")) {
                    parent = item.getName();
                }

                localParentDir = item.getAbsolutePath();
                uploadDirectory(ftpClient, remoteDirPath, localParentDir,
                        parent);
            }
        }
    }
}

Every time I am uploading the files (files are of different types like .iso, .dat etc), The first few files (upload sequence is like first few hundred files are smaller i.e less than few MBs then Last 10 files are more than 1 GB big)will be successfully uploaded (i.e all smaller files and 2 of the last 10 files) but when it starts uploading the third big file I get following exception.

SEVERE: null
org.apache.commons.net.io.CopyStreamException: IOException caught while copying.
    at org.apache.commons.net.io.Util.copyStream(Util.java:134)
    at org.apache.commons.net.ftp.FTPClient._storeFile(FTPClient.java:653)
    at org.apache.commons.net.ftp.FTPClient.__storeFile(FTPClient.java:624)
    at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:1976)

Answer 1

The CopyStreamException has a "cause" exception. Check that using the .getCause(), to see what went wrong.

See the Util.copyStream method:

public static final long copyStream(InputStream source, OutputStream dest,
                                    int bufferSize, long streamSize,
                                    CopyStreamListener listener,
                                    boolean flush)
throws CopyStreamException
{
    int bytes;
    long total = 0;
    byte[] buffer = new byte[bufferSize >= 0 ? bufferSize : DEFAULT_COPY_BUFFER_SIZE];

    try
    {
        while ((bytes = source.read(buffer)) != -1)
        {
             ....
        }
    }
    catch (IOException e)
    {
        throw new CopyStreamException("IOException caught while copying.",
                                      total, e);
    }

    return total;
}

Somewhere in your uploadSingleFile function, do

try
{
     ftpClient.storeFile(...)
}
catch (Exception e)
{
     e.printStackTrace();
     if (e.getCause() != null)
     { 
         e.getCause().printStackTrace();
     }
}

I do not know Java, so the code may not be 100% correct.

See also Getting full string stack trace including inner exception.