Standard Input Stream hangs when paramiko throws 'No existing session' exception

Question

I am using java to launch a python script using the method below:

private String[] runScript(String args) {
        ArrayList<String> lines = new ArrayList<String>();
        try {
            String cmd = "python py/grabber/backdoor.py " + args;
            System.out.println(cmd);
            Runtime run = Runtime.getRuntime();
            Process pr = run.exec(cmd);
            InputStream stdout = pr.getInputStream();
            BufferedReader stdInput = new BufferedReader(new InputStreamReader(stdout, StandardCharsets.UTF_8));
            String line;
            System.out.println("Starting to read output:");
            while ((line = stdInput.readLine()) != null) {
                if(!pr.isAlive()) {
                    break;
                }
                System.out.println("" + line);
                lines.add(line);
            }
            stdInput.close();
        } catch (Exception e) {
            System.out.println("Exception in reading output" + e.toString());
        }
        return lines.toArray(new String[lines.size()]);
    }

The python script starts executing the below method for a list of devices on a new child thread for each device:

def run_command(device, retry):
    try:
        if(retry != 0):
            save_log('Retrying '+device['ip']+'...')
        SSHClient = netmiko.ssh_dispatcher("cisco_ios")
        ssh_connection = SSHClient(**device)

        ruckus = ("ruckus" in str(device['device_type']))

        output = ssh_connection.send_command('show run | inc hostname')
        if(not ruckus):
            output = output.split('\n')[0]
            hostname = output.split(' ')[1] + '-' + today
        else:
            hostname = output.split(' ')[1] + '-' + today

        msg = (''+device['ip']+':Retry Success! Hostname:'+hostname+'\n',''+device['ip']+':hostname: ' + hostname + '\n')[retry == 0]
        save_log(msg)
        return True
    except Exception as e:
        if("Authentication" in str(e)):
            if(config.exe_i):
                save_log('\nAuthentication failed! Consider resetting credentials\n')
                if(not config.using_gui):
                    val = input("Continue with list? [y]/[n]: ")
                    if("n" in val):
                        return False
                else:
                    return True
            else:
                msg = '\nSwitch: ' + device['ip'] + '\nLogin Failure'
                save_error(msg)
                return True
        elif("SSH protocol banner" in str(e) and retry <= 3):
            retry += 1
            save_log('\nSwitch: ' + device['ip'] + ' - SSH Banner.. Attempt: '+str(retry))
            return run_command(device, retry=retry)
        elif("conn_timeout" in str(e) and retry <= 3):
            retry += 1
            save_log('\nSwitch: ' + device['ip'] + ' - Connection Timeout.. Attempt: '+str(retry))
            return run_command(device, retry=retry)
        error = 'error @ switch: ' + device['ip'] + ' with error: \n' + str(e)
        save_error(error)
        return True

When the python script gets this particular exception,

Paramiko: 'No existing session' error: try increasing 'conn_timeout' to 10 seconds or larger.

the script will simply retry to connect and proceed as normal as intended, but the java program begins to hang at the while loop despite the fact that the python script finishes executing, successfully. Why is it that when this exception is thrown the standard input stream in the while loop begins to hang?

Additional Info:

1.) When taking threads out of the scenario the while loop will still hang if the 'no existing session' exception is thrown.

2.) Another common exception that causes a retry ssh attempt, "SSH protocol banner", does not cause the while loop to hang. Therefore, I am concluding that the problem is not the way the exception itself is being handled in my program stack.

David Pace · Accepted Answer · 2021-07-03T00:46:10.660

The problem is with the Java segment, and the solution is to not only read the input stream, but also the output stream because the process will hang if both are not emptied. In order to do this properly, each stream needs to be read on a separate thread. ProcessBuilder is a better alternative because it allows you to combine the input and output stream into one and then only the main thread is necessary.

This stack post is a reference to the more canonical question for this concept: Java Process with Input/Output Stream

Standard Input Stream hangs when paramiko throws 'No existing session' exception

1 Answers1