Can't close an SSH connection opened with Popen

Question

I created a class method (this will only run on Linux) that sends a list of commands to a remote computer over SSH and returns the output using subprocess.Popen:

def remoteConnection(self, list_of_remote_commands):
    ssh = subprocess.Popen(["ssh", self.ssh_connection_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,universal_newlines=True, bufsize=0)

    # send ssh commands to stdin
    for command in list_of_remote_commands:
        ssh.stdin.write(command + "\n")
        ssh.stdin.close()

    output_dict = {'stdin': list(ssh.stdin), 'stdout': list(ssh.stdout), 'stderr': list(ssh.stderr)}

    return output_dict

Whilst I'm still getting to grips with the subprocess module I'd read quite a bit about Popen and no one ever mentioned closing it (SSH Connection with Python 3.0, Proper way to close all files after subprocess Popen and communicate, https://docs.python.org/2/library/subprocess.html) so I assumed that that wasn't a problem.

However when testing this out in ipython outside of a function I noticed that the variable ssh still seemed active. I tried closing ssh.stdin, ssh.stdout and ssh.stderr and even ssh.close(), ssh.terminate() and ssh.kill() but nothing seemed to close it. I thought perhaps it doesn't matter but my function will be called many times for months or even years so I don't want it to spawn a new process everytime it is run otherwise I'm going to quickly use up my maximum processes limit. So I use ssh.pid to find the PID and look it up using ps aux | grep PID and it's still there even after doing all of the above.

I also tried:

with subprocess.Popen(["ssh", self.ssh_connection_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,universal_newlines=True, bufsize=0) as shh:

instead of:

ssh = subprocess.Popen(["ssh", self.ssh_connection_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,universal_newlines=True, bufsize=0)

I also remember solving a similar problem a while back using ssh -T but even:

ssh = subprocess.Popen(["ssh", "-T",  self.ssh_connection_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,universal_newlines=True, bufsize=0)

Didn't work.

I'm sure I would have found something about closing Popen if I needed to but then why is the process still open on my computer - can anyone help me understand what's going on here?

opening an `ssh` connection with `popen` sounds like a bad practice, have you looked into `ssh_decorate`, or `paramiko` ? — Uri Goren, May 04 '18 at 14:08

Jean-François Fabre · Accepted Answer · 2018-04-07T08:36:10.857

In your case, you have a deadlock here:

output_dict = {'stdin': list(ssh.stdin), 'stdout': list(ssh.stdout), 'stderr': list(ssh.stderr)}

Mostly because list(ssh.stdin) blocks forever: trying to read standard input of a process doesn't work (there's also an extra risk because you redirected both standard output & error to different pipes without using threading to consume them)

You mean to use ssh.communicate, passing the whole input as argument. Simply do:

command_input = "".join(["{}\n".format(x) for x in list_of_remote_commands])
output,error = ssh.communicate(command_input)  # may need .encode() for python 3
return_code = ssh.wait()

then

output_dict = {'stdin': list_of_commands, 'stdout': output.splitlines(), 'stderr': error.splitlines()}

I may add that in the particular ssh case, using paramiko module is better (python paramiko ssh) and avoids using subprocess completely.

the `ssh.communicate` command throws the error: `AttributeError: 'list' object has no attribute 'encode'` Perhaps you meant something like: `command_input = "\n".join(list_of_remote_commands) + "\n"` ? — ojunk, Apr 06 '18 at 17:39

Can't close an SSH connection opened with Popen

1 Answers1