Download all files in directory with SCP protocol using JSch

Question

I've been trying to retrieve a list of files that have file names that don't contain the string "-ingested-" via SCP (my manager doesn't want to use SFTP) from a remote server.

The files are zip files with at least one .txt file in them and a related .txt.count file.

So once I did some research on java libraries that allow retrieval of files via SCP, I broke the task down as follows:

1. Connect to the remote server (key authentication only) using JSch - done

2. Run a command that retrieves a specific file via SCP - done

3. Run a command that retrieves all the files in the specified directory via SCP - stuck on this, can't find any examples of doing this in JSch

4. Run a command that retrieves all the files in the specified directory that have file names that do NOT contain the string "-ingested-" in it

5. Rename all the files on the remote server that do not have the string "-ingested-" in their file name, and add the string "-ingested-" to their file name. This is so that when I go to retrieve the files from the server a couple of hours later, I don't retrieve the ones I have already read in.

I'm currently stumped on step 3 as I don't seem to be able to find a example of this anywhere.

Below is the code for step 1 and 2:

final JSch jsch = new JSch();

final String authenticationKeyFilePath = "/var/myprivatekey.rsa";

jsch.addIdentity(authenticationKeyFilePath);

final String knownHostFilePath = "/home/user1/.ssh/known_hosts";

jsch.setKnownHosts(knownHostFilePath);

final String userName = "p";
final String host = "example.com";
final int port = 22;

final Session session = jsch.getSession(userName, host, port);

session.connect();

final String channelType = "exec";

final Channel channel = session.openChannel(channelType);

final String query = "scp -f /home/download/50347_SENT_20170614_025807.txt.count";

((ChannelExec)channel).setCommand(query);

// Todo: Dispose of these streams if necessary.
final OutputStream outputStream = channel.getOutputStream();
final InputStream inputStream = channel.getInputStream();

channel.connect();

byte[] buffer = new byte[1024];
buffer[0] = 0;

final int outputStreamOffset = 0;
final int outputStreamLength = 1;
outputStream.write(buffer, outputStreamOffset, outputStreamLength);
outputStream.flush();

while (true) {
    final int c = checkAck(inputStream);

    if (c != 'C') break;

    // read '0644 '
    inputStream.read(buffer, 0, 5);

    long filesize = 0L;

    while (true) {
        if (inputStream.read(buffer, 0, 1) < 0) break;

        if (buffer[0] == ' ') break;
        filesize = filesize * 10L + (long)(buffer[0] - '0');
    }

    String file = null;

    for (int i = 0; ; i++) {
        inputStream.read(buffer, i, 1);

        if (buffer[i] == (byte)0x0a){
            file = new String(buffer, 0, i);
            break;
        }
    }

    // send '\0'
    buffer[0] = 0;
    outputStream.write(buffer, 0, 1);
    outputStream.flush();

    // read a content of lfile
    FileOutputStream fos = new FileOutputStream(file);
    int foo;

    while (true) {
        if (buffer.length < filesize) foo = buffer.length;
        else foo = (int)filesize;
        foo = inputStream.read(buffer, 0, foo);

        if (foo < 0){
            // error
            break;
        }

        fos.write(buffer, 0, foo);
        filesize -= foo;

        if (filesize == 0L) break;
    }

    final String textFromDownloadedFile = fos.toString();  

    fos.close();
    fos = null;

    if (checkAck(inputStream) != 0) System.exit(0);

    // send '\0'
    buffer[0] = 0;
    outputStream.write(buffer, outputStreamOffset, outputStreamLength);
    outputStream.flush();
}

session.disconnect();

System.exit(0);

So in an ideal world, I would change the query string to something that matches what I want in step 4 and read them into a string list or whatever is most efficient, then I can store them and move onto step 5. Any help is much appreciated.

p.s. If there is an easier to use Java SCP library out there with a corporation friendly licence, please let me know!

Answer 1

Use a command like:

scp -r -d -f /remote/path/*

And then loop over the files that the server sends to you. What your code seems to do already.

---

Note that I'm afraid you will have troubles implementing the step 4 using the SCP protocol. The "NOT contain" condition is troublesome.

Use SFTP! The SCP is legacy obsolete protocol.

Of course, what you can do is to execute ls shell command. Parse the results to find list of all files. And then filter them locally.

---

Some related questions:

• Explanation for SCP protocol implementation in JSch library
• JSch - Explaining SCP stream reading
• JSch SCP file transfer using "exec" channel