I implemented java code to upload files to server with org.apache.commons.net.ftp.FTPClient For multiple files the ftp upload speed is very slow. How can I improve the speed.
-Change library? What is the powerful FTP client class library for uploading multiple files?
-Use multiple threads? How can I implement ftp upload function with multiple thread? Could someone show me an example? I am a new for multiple threading programming.
After I read all answer, I try to change my code and test it.
Following is a sample FTPClient code:
// create instance of FTPClient
FTPClient ftp = new FTPClient();
ftp.setControlEncoding("UTF-8");
ftp.setDefaultTimeout(30000);
// connect to server
try
{
ftp.connect("10.1.1.1", 990);
}
catch(Exception e)
{
System.out.println("Cannot connect to server");
return;
}
// login to server
if (!ftp.login("username", "password"))
{
ftp.logout();
System.out.println("Cannot login to server");
return;
}
try
{
ftp.setFileTransferMode(FTP.BINARY_FILE_TYPE);
ftp.enterLocalPassiveMode();
// ftp.setBufferSize(0); <-- someone suggest me to set buffer size to 0, but it throw error sometime.
}
catch(Exception e)
{
}
// create directory on server
// dirs is list of required directories on server
for (String dir : dirs)
{
try
{
ftp.makeDirectory(dir);
}
catch(IOException e)
{
}
}
// files is a map of local file and string of remote file
// such as
// file on client is "C://test/a.txt"
// location on server is "/test/a.txt"
for (Map.Entry<File, String> entry : files.entrySet())
{
File localFile = entry.getKey();
String remoteFile = entry.getValue();
FileInputStream input = null;
try
{
input= new FileInputStream(localFile);
ftp.storeFile(remoteFile, input);
}
catch (Exception e)
{
try
{
ftp.deleteFile(remoteFile);
}
catch (IOException e1)
{
}
}
finally
{
if (input != null)
{
try
{
input.close();
}
catch (IOException e)
{
}
}
}
}
// disconnect
if (ftp != null && ftp.isConnected())
{
try
{
ftp.disconnect();
}
catch (IOException f)
{
// do nothing
}
}
When I uploaded 1050 files (each file is about 1-20 KB), it took about 49406 - 51000 millisec (This is upload time only). I would like to improve the speed.
Some people suggest me to use ftp4j, but when I test the library with 1050 files, the upload speed of ftp4j is slowly than FTPClient about 10000 millisec. it took about 60000 millisec.
Following is sample ftp4j code:
// create instance of FTPClient
FTPClient ftp = new FTPClient();
ftp.setCharset("UTF-8");
// connect to server
try
{
ftp.connect("10.1.1.1", 990);
}
catch(Exception e)
{
System.out.println("Cannot connect to server")
return;
}
// login to server
try
{
ftp.login("username", "password");
}
catch (Exception e)
{
try
{
ftp.logout();
}
catch (Exception e1)
{
}
System.out.println("Cannot login to server")
return;
}
try
{
ftp.setType(FTPClient.TYPE_BINARY);
ftp.setPassive(true);
}
catch(Exception e)
{
}
// create directory on server
// dirs is list of required directories on server
for (String dir : dirs)
{
try
{
ftp.createDirectory(dir);
}
catch (Exception e)
{
}
}
// files is a map of local file and string of remote file
// such as
// file on client is "C://test/a.txt"
// location on server is "/test/a.txt"
for (Map.Entry<File, String> entry : files.entrySet())
{
final File localFile = entry.getKey();
final String remoteFile = entry.getValue();
BufferedInputStream input = null;
boolean success = false;
try
{
input = new BufferedInputStream(new FileInputStream(localFile));
// ftp.upload(localFile); <-- if I use ftp.upload(File), it will took more time.
ftp.upload(remoteFile, input, 0, 2048, new MyTransferListener());
success = true;
}
catch (Exception e)
{
}
finally
{
if (input != null)
{
try
{
input.close();
}
catch (IOException e)
{
}
}
if (!success)
{
try
{
ftp.deleteFile(remoteFile);
}
catch (Exception e)
{
}
}
}
}
// disconnect
if (ftp != null && ftp.isConnected())
{
try
{
ftp.disconnect();
}
catch (IOException f)
{
// do nothing
}
}
I try to use multiple threads.
Following is multiple threading code:
final CountDownLatch latch = new CountDownLatch(files.size());
ExecutorService pool = Executors.newFixedThreadPool(10);
for (Map.Entry<File, String> entry : files.entrySet())
{
final File localFile = entry.getKey();
final String remoteFile = entry.getValue();
pool.execute(new Runnable() {
public void run()
{
FileInputStream input = null;
try
{
input= new FileInputStream(localFile);
ftp.storeFile(remoteFile, input);
}
catch (Exception e)
{
try
{
ftp.deleteFile(remoteFile);
}
catch (IOException e1)
{
}
}
finally
{
if (input != null)
{
try
{
input.close();
}
catch (IOException e)
{
}
}
latch.countDown();
}
}
});
}
try
{
// waiting for all threads finish
// see: http://stackoverflow.com/questions/1250643/how-to-wait-for-all-threads-to-finish-using-executorservice
latch.await();
}
catch(Exception e)
{
}
Is it correct? It work correctly but it cannot improve speed. it took about 49000 - 51000 millisec same as the code without thread.
I test the speed with intranet. It will take more time for internet.
How should I do for improve upload speed?
