How to split a large file into multiple files on SFTP server using JSch in Java

Question

I have a very large file around 500 GB file in an SFTP location. I need to split this file into smaller files and needs to place the same in the subdirectory. I am trying the JSch utility to connect to SFTP server and split the same but only one file is getting created and no further files are getting created.

JSch jsch = new JSch();
//create SSH connection
String host = "hostname";
String user = "username";
String password = "password";
Session session = jsch.getSession(user, host, 22);
java.util.Properties config = new java.util.Properties(); 
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.setPassword(password);
session.connect();
long linesWritten = 0;
int  count = 1;
int  linesPerSplit = 3000;
String inputFilePath  = "/basedirector/largefile.txt";
String outputFolderPath = "/subfolder/smallfiles";
try {
    Channel channel = session.openChannel("sftp");
    channel.connect(60000);
    ChannelSftp sftpChannel = (ChannelSftp) channel;
    File inputFile = new File(inputFilePath);
    InputStream stream = sftpChannel.get(inputFilePath);
    BufferedReader br = new BufferedReader(new InputStreamReader(stream));
    String line = br.readLine();
    String outfileName = outputFolderPath + "\\" + "SmallFile";  
    Channel channel1 = session.openChannel("sftp");    
    channel1.connect();
    ChannelSftp sftpChannel1 = (ChannelSftp) channel1;
    sftpChannel1.cd(outputFolderPath);
    while (line != null) {
        String outfileName1 = "SmallFile"+ "_" + count + ".txt";
        System.out.println("is session connected" +session.isConnected());
        OutputStream outFile = sftpChannel1.put(outfileName1);
        OutputStreamWriter writer = new OutputStreamWriter(outFile);
        writer.write("Hello World");
        writer.close();
        while (line != null && linesWritten < linesPerSplit) {
            String str_Content=line + System.lineSeparator();
            System.out.println(linesWritten);
            line = br.readLine();
            linesWritten++;
        }
        sftpChannel1.exit();
        sftpChannel1.disconnect();
        sftpChannel1.connect();
        linesWritten = 0;//next file
        count++;//next file count
    }
   // channel.disconnect();
    br.close();
    //session.disconnect();
    inputFile.delete();
    System.out.println("end of the process");
} catch (Exception e) {
    e.printStackTrace();
}

score 0 · Answer 1 · answered Jun 17 '21 at 05:30

If you need to split the file in-place on the server (without transferring it to the local machine and back), you basically cannot use SFTP, unless you have a server that supports that, what most server don't.

That's covered here:
Using SFTP in Java, How do I transfer a file from one folder to another?

How to split a large file into multiple files on SFTP server using JSch in Java

1 Answers1