How to use Java JSch library to read remote file line by line?

Question

I am trying to use Java to read a file line by line, which is very simple (there are multiple solutions for this on stackoverflow.com), but the caveat here is that the file is located on a remote server, and it is not possible to get a local copy (it is a massive collection of millions of Amazon reviews in a single .txt file).

JSch comes with two example classes that copy files to and from remote hosts, namely ScpTo and ScpFrom. I am interested in reading the file from the remote host line by line; ScpFrom would try to copy the whole thing into a local file, which would take ages.

Here is a link to ScpFrom: http://www.jcraft.com/jsch/examples/ScpFrom.java.html

I would try to cargo cult the code there and then modify it to read a remote file line by line rather than write to a local file, but most of the code is Greek to me once the author declares a byte array and starts reading bytes from the remote file. I'll admit this is something I have almost no understanding of; BufferedReader provides a much higher level interface. Essentially I want to do this: How to read a large text file line by line using Java?

except using a BufferReader that can also read remote files line by line, if provided the host name and user credentials (password, etc.), i.e. RemoteBufferReader?

This is the test code I've written; how do I read in the remote file line by line using JSCh?

public class test2
 {
    static String user = "myusername";
    static String host = "user@remotehost";
    static String password = "mypasswd";
    static String rfile = "/path/to/remote/file/on/remote/host";
    public static void main(String[] args) throws FileNotFoundException, IOException, JSchException
    {
        JSch jsch=new JSch();
        Session session=jsch.getSession(user, host, 22);
        session.setPassword(password);
        session.connect();
        // exec 'scp -f rfile' remotely
        String command="scp -f "+rfile;
        Channel channel=session.openChannel("exec");
        ((ChannelExec)channel).setCommand(command);

        // get I/O streams for remote scp
        OutputStream out=channel.getOutputStream();
        channel.connect()
        //no idea what to do next

    }
 }

I would used `cat` (on the remote host) rather than `scp`. Then just read the stdout of the command line-by-line. — Rob, Sep 04 '14 at 05:10
You can open an inputstream over sftp with jsch and read that line by line — nablex, Sep 04 '14 at 06:13

score 26 · Accepted Answer · edited Sep 20 '21 at 17:53

26

To manipulate files through ssh, you're better off using sftp than scp or pure ssh. Jsch has built-in support for sftp. Once you've opened a session, do this to open an sftp channel:

ChannelSftp sftp = (ChannelSftp) session.openChannel("sftp");

Once you've opened an sftp channel, there are methods to read a remote file which let you access the file's content as an InputStream. You can convert that to a Reader if you need to read line-by-line:

InputStream stream = sftp.get("/some/file");
try {
    BufferedReader br = new BufferedReader(new InputStreamReader(stream));
    // read from br
} finally {
    stream.close();
}

Using try with resources syntax, your code might look more like this:

try (InputStream is = sftp.get("/some/file");
     InputStreamReader isr = new InputStreamReader(is);
     BufferedReader br = new BufferedReader(isr)) {
    // read from br
}

edited Sep 20 '21 at 17:53

ouzture

76
1
12

answered Sep 05 '14 at 21:09

Kenster

23,465
21
80
106

Perfect, this did exactly what I needed. Thank you! – user3898238 Sep 05 '14 at 22:29
1

This is standard sftp functionality. You don't need to run commands using "exec" channel to read/transfer files. – Vishnu Prasad Kallummel Sep 15 '14 at 10:53
I'm trying to open an sftp for remote windows machine and get the content, but I'm getting filenotfound exception. The file is existing with the path specified. Is there any other thinks I need to do for windows? InputStream stream = channel.get("C:\\1.xml"); – artos Jul 28 '15 at 09:23
@artos http://stackoverflow.com/a/31519959/13317. If you're still having trouble, you should click on the "Ask a question" button at the top of this page and ask a new question. – Kenster Jul 28 '15 at 11:11

score 7 · Answer 2 · answered Oct 04 '17 at 07:56

JSch library is the powerful library that can be used to read file from SFTP server. Below is the tested code to read file from SFTP location line by line

        JSch jsch = new JSch();
        Session session = null;
        try {
            session = jsch.getSession("user", "127.0.0.1", 22);
            session.setConfig("StrictHostKeyChecking", "no");
            session.setPassword("password");
            session.connect();

            Channel channel = session.openChannel("sftp");
            channel.connect();
            ChannelSftp sftpChannel = (ChannelSftp) channel;

            InputStream stream = sftpChannel.get("/usr/home/testfile.txt");
            try {
                BufferedReader br = new BufferedReader(new InputStreamReader(stream));
                String line;
                while ((line = br.readLine()) != null) {
                    System.out.println(line);
                }

            } catch (IOException io) {
                System.out.println("Exception occurred during reading file from SFTP server due to " + io.getMessage());
                io.getMessage();

            } catch (Exception e) {
                System.out.println("Exception occurred during reading file from SFTP server due to " + e.getMessage());
                e.getMessage();

            }

            sftpChannel.exit();
            session.disconnect();
        } catch (JSchException e) {
            e.printStackTrace();
        } catch (SftpException e) {
            e.printStackTrace();
        }

Please refer the blog for whole program.

How to use Java JSch library to read remote file line by line?

2 Answers2

Linked