I wrote a program in Java that connects to an FTPS server, loops over all directories within a certain parent looking for files whose names match a given regexp, and then retrieve these files to the local machine. I'm using Apache Commons Net 3.6 and Java 8. Here's how I connect and get to the list of folders I need to iterate over. fileType is a String input parameter and can be null. fileNameRegExp is also a String input param, which holds the regexp that will filter only the files with the proper naming:
String server = "<here I type the dot-separated IP address for the FTPS server>";
int port = <port number>;
String user = "username";
String pass = "********";
String folder = "C:\\Target\\InputFiles\\";
String protocol = "TLS";
int timeoutInMillis = 30000;
FTPSClient sftpClient = new FTPSClient(protocol, true);
int reply ;
try {
sftpClient.setDataTimeout(timeoutInMillis);
sftpClient.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out)));
FTPClientConfig clientConfig = new FTPClientConfig(FTPClientConfig.SYST_NT);
sftpClient.configure(clientConfig);
sftpClient.connect(server, port);
sftpClient.setKeepAlive(Boolean.TRUE);
sftpClient.login(user, pass);
reply = sftpClient.getReplyCode();
System.out.println("INFO: ftp login status = <"+reply+">.");
sftpClient.enterLocalPassiveMode();
sftpClient.setFileType(FTP.BINARY_FILE_TYPE);
sftpClient.execPBSZ(0);
sftpClient.execPROT("P");
sftpClient.changeWorkingDirectory("/main_folder");
reply = sftpClient.getReplyCode();
System.out.println("INFO: ftp CWD command on /main_folder status = <"+reply+">.");
List<String> ftpDirectories = new ArrayList<>();
if( fileType!= null && !fileType.isEmpty() ) {
ftpDirectories.add("/main_folder/"+fileType);
} else {
FTPFile[] ftpDirList = sftpClient.listDirectories();
for( FTPFile dir : ftpDirList)
ftpDirectories.add("/main_folder/"+dir.getName());
}
for( String workDirectory : ftpDirectories) {
sftpClient.changeWorkingDirectory(workDirectory);
FTPFile[] ftpFiles = Arrays.stream(sftpClient.listFiles())
.filter(file -> file.getName().matches(fileNameRegExp))
.toArray(FTPFile[]::new);
for (FTPFile ftpFile : ftpFiles) {
(...here's the file transfer logic for each FTPFile found...)
}
}
}catch(Exception ex){
}
Most of the folders won't have matching files and that's expected, but it could be that I find a bunch in a few or all of the subfolders in /main_folder. For testing purposes I added 2 matching files, each one in a different subfolder. Here's a sample output to the console from a test run:
220 Microsoft FTP Service
USER username
331 Password required
PASS ********
230 User logged in.
INFO: ftp login status = <230>.
TYPE I
200 Type set to I.
PBSZ 0
200 PBSZ command successful.
PROT P
200 PROT command successful.
CWD /main_folder
250 CWD command successful.
INFO: ftp CWD command on /main_folder status = <250>.
PASV
227 Entering Passive Mode (GG,XX,TT,VV,N,LL).
[Replacing site local address XX.XX.XX.XX with YY.YY.YY.YY]
LIST
125 Data connection already open; Transfer starting.
226 Transfer complete.
CWD /main_folder/subfolder_1
250 CWD command successful.
PASV
227 Entering Passive Mode (GG,XX,TT,VV,Z,KK).
[Replacing site local address XX.XX.XX.XX with YY.YY.YY.YY]
LIST
125 Data connection already open; Transfer starting.
226 Transfer complete.
CWD /main_folder/subfolder_2
250 CWD command successful.
(...pretty much the same for other folders with no matching files...)
When it finds the first matching file, it goes on like this:
CWD /main_folder/subfolder_winner_1
250 CWD command successful.
PASV
227 Entering Passive Mode (GG,XX,TT,VV,W,RR).
[Replacing site local address XX.XX.XX.XX with YY.YY.YY.YY]
LIST
125 Data connection already open; Transfer starting.
226 Transfer complete.
PASV
227 Entering Passive Mode (GG,XX,TT,VV,M,OO).
[Replacing site local address XX.XX.XX.XX with YY.YY.YY.YY]
RETR matching_file_1.txt
125 Data connection already open; Transfer starting.
So far so good. The file gets read and copied over the socket to the local folder as it should. Now, when the code finds the other file I get in trouble. The call to sftpClient.listFiles() returns empty. If, for debugging purposes, I try and check if the returned array is empty and try to run a 2nd call to sftpClient.listFiles(), I get an exception: java.net.ConnectException: Connection refused: connect
The funny thing is that right before that I call sftpClient.isConnected() and sftpClient.isAvailable() and both return true
At this point all I care about is making the code list and transfer the 2nd file no matter what, so I change the code to divert the flow when it reaches that subfolder with the 2nd file. In the new flow, I logout, disconnect, do all that FTPSClient setup again, as disconnect resets some stuff in the object, reconnect, login again and prestoooo. It reads and transfers the file. Of course, it's a cheat. I knew where it was failing, placed a road detour with disconnection and reconnection, so it's no solution. And here's where I'm completely in the dark and out of ideas. I ran the checks I knew to make sure the socket was still available and working and that the program was still connected to the remote server and it still failed. There does seem to be something up with the connection/socket, but unless there's a test I can run so I know it went stale, I'll never know when disconnection/reconnection is required for the program to go on.
Just as one last remark, the only way I have to check on the FTPS server is connecting through FileZilla, using the exact same username, passwd, server ip and port, and it works fine this way. It lists me the directories in /main_folder, I can see and rename all files in the tree, and can transfer files back and forth. I have no means of connecting to it using a console, or calling sftp at cmd or bash.
I welcome any ideas you might have as to how to solve this or to gain more insight into the bug.
Thank you in advance and kind regards
UPDATE:
I'd like to thank @AllinProgram for his suggestion, but unfortunately the result was the same. Here's my override:
@Override
protected void _prepareDataSocket_(final Socket socket) throws IOException {
if(socket instanceof SSLSocket) {
// Control socket is SSL
final SSLSession session = ((SSLSocket) _socket_).getSession();
final SSLSessionContext context = session.getSessionContext();
//context.setSessionCacheSize(0);
try {
final Field sessionHostPortCache = context.getClass().getDeclaredField("sessionHostPortCache");
sessionHostPortCache.setAccessible(true);
final Object cache = sessionHostPortCache.get(context);
final Method method = cache.getClass().getDeclaredMethod("put", Object.class, Object.class);
method.setAccessible(true);
String key = String.format("%s:%s", socket.getInetAddress().getHostName(), String.valueOf(socket.getPort())).toLowerCase(Locale.ROOT);
method.invoke(cache, key, session);
key = String.format("%s:%s", socket.getInetAddress().getHostAddress(), String.valueOf(socket.getPort())).toLowerCase(Locale.ROOT);
method.invoke(cache, key, session);
}
catch(NoSuchFieldException e) {
// Not running in expected JRE
System.out.println("No field sessionHostPortCache in SSLSessionContext. Exception: " + e);
}
catch(Exception e) {
// Not running in expected JRE
System.out.println(e.getMessage());
}
}
}
In my main method class I call System.setProperty("jdk.tls.useExtendedMasterSecret", "false");
There's one thing I noticed though, after further debugging. The listFiles() actually triggers a new data connection for the LIST command (call to protected Socket _openDataConnection_(String command, String arg) in the parent class, which triggers a call to public int pasv() within FTP class). When there're no matching files in a directory the code issues a CWD, a PASV and a LIST. When there're files matching, it's CWD, PASV, LIST, PASV, RETR. And then, for the directory that follows, when listFiles() gets triggered, the PASV command returns a 250 reply. At this point the last command issued was a CWD, also with a 250 reply. The method called for issuing FTP commands is as below:
public int sendCommand(String command, String args) throws IOException {
if (this._controlOutput_ == null) {
throw new IOException("Connection is not open");
} else {
String message = this.__buildMessage(command, args);
this.__send(message);
this.fireCommandSent(command, message);
this.__getReply();
return this._replyCode;
}
}
A PASV should return 227, but this call gets me a 250. It's almost as if it ran, but the reply code got stuck in the CWD reply code, which was 250. That causes openDataConnection to return a null for a socket and private FTPListParseEngine initiateListParsing(FTPFileEntryParser parser, String pathname) returns an FTPListParseEngine object that doesn't read the server list of files in the current work directory. That's why listFiles() returns an empty FTPFile array.
I tried overriding sendCommand in order to get a PASV with 227 (issuing a new PASV right after the previous one), but when the code tries to connect to the socket, I get the java.net.ConnectException: Connection refused, almost as if the server isn't listening to that port, but it was the server that returned those socket details for the data socket, right? I mean, that's the result of a PASV: the server telling the client where to go to retrieve the data. Why would it refuse the connection?
I hope this makes sense to somebody, cause a 250 PASV doesn't. I'm quite lost here. :o/
