python subprocess communicate freezes

Question

I have the following python code that hangs :

cmd = ["ssh", "-tt", "-vvv"] + self.common_args
cmd += [self.host]
cmd += ["cat > %s" % (out_path)]
p = subprocess.Popen(cmd, stdin=subprocess.PIPE,
                         stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate(in_string)

It is supposed to save a string (in_string) into a remote file over ssh.

The file is correctly saved but then the process hangs. If I use

cmd += ["echo"] instead of
cmd += ["cat > %s" % (out_path)]

the process does not hang so I am pretty sure that I misunderstand something about the way communicate considers that the process has exited.

do you know how I should write the command so the the "cat > file" does not make communicate hang ?

I think this [post](http://stackoverflow.com/a/19202567/1982962) can help, it's an example of how to write to a remote file using SSH — Kobi K, Nov 28 '13 at 11:04
Slight tangent, but rather than using an SSH process to do this, have you considered something like [SSHFS](http://fuse.sourceforge.net/sshfs.html) ? This would mean that you'd only need to worry about writing to a file, rather than maintaining all of this — Andrew Walker, Nov 28 '13 at 11:16

score 1 · Accepted Answer · answered Nov 29 '13 at 22:17

-tt option allocates tty that prevents the child process to exit when .communicate() closes p.stdin (EOF is ignored). This works:

import pipes
from subprocess import Popen, PIPE

cmd = ["ssh", self.host, "cat > " + pipes.quote(out_path)] # no '-tt'
p = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate(in_string)

You could use paramiko -- pure Python ssh library, to write data to a remote file via ssh:

#!/usr/bin/env python
import os
import posixpath
import sys
from contextlib import closing

from paramiko import SSHConfig, SSHClient

hostname, out_path, in_string = sys.argv[1:] # get from command-line 

# load parameters to setup ssh connection
config = SSHConfig()
with open(os.path.expanduser('~/.ssh/config')) as config_file:
    config.parse(config_file)
d = config.lookup(hostname)

# connect
with closing(SSHClient()) as ssh:
    ssh.load_system_host_keys()
    ssh.connect(d['hostname'], username=d.get('user'))
    with closing(ssh.open_sftp()) as sftp:
        makedirs_exists_ok(sftp, posixpath.dirname(out_path))
        with sftp.open(out_path, 'wb') as remote_file:
            remote_file.write(in_string)

where makedirs_exists_ok() function mimics os.makedirs():

from functools import partial
from stat import S_ISDIR

def isdir(ftp, path):
    try:
        return S_ISDIR(ftp.stat(path).st_mode)
    except EnvironmentError:
        return None

def makedirs_exists_ok(ftp, path):
    def exists_ok(mkdir, name):
        """Don't raise an error if name is already a directory."""
        try:
            mkdir(name)
        except EnvironmentError:
            if not isdir(ftp, name):
                raise

    # from os.makedirs()
    head, tail = posixpath.split(path)
    if not tail:
        assert path.endswith(posixpath.sep)
        head, tail = posixpath.split(head)

    if head and tail and not isdir(ftp, head):
        exists_ok(partial(makedirs_exists_ok, ftp), head)  # recursive call

    # do create directory
    assert isdir(ftp, head)
    exists_ok(ftp.mkdir, path)

but why does it work with -tt + echo ? your explanation seems to imply that it should not work with echo either !? — Jerome WAGNER, Nov 30 '13 at 09:35
@JeromeWAGNER: `echo` doesn't expect input on stdin. Try any program that doesn't exit until all input (stdin) is consumed. Though I do not understand it fully either. It might be a bug somewhere in the chain with closing pseudo-tty file descriptors. It is platform dependant e.g., here's [code that uses `pty` (it is how I imagine ssh launches remote child processes (I can be completely wrong about it))](http://stackoverflow.com/a/12471855/4279) — jfs, Nov 30 '13 at 10:01

score 0 · Answer 2 · answered Nov 28 '13 at 11:50

0

It makes sense that the cat command hangs. It is waiting for an EOF. I tried sending an EOF in the string but couldn't get it to work. Upon researching this question, I found a great module for streamlining the use of SSH for command line tasks like your cat example. It might not be exactly what you need for your usecase, but it does do what your question asks.

Install fabric with

pip install fabric

Inside a file called fabfile.py put

from fabric.api import run

def write_file(in_string, path):
    run('echo {} > {}'.format(in_string,path))

And then run this from the command prompt with,

fab -H username@host write_file:in_string=test,path=/path/to/file

answered Nov 28 '13 at 11:50

William Denman

3,046
32
34

thanks for the idea but I am trying to patch an existing project (ansible) and the dependency on fabric would not be accepted. – Jerome WAGNER Nov 29 '13 at 15:02
welcome. but yes, I figured it might not be suited for your needs. Did Kobi K's answer work for you? – William Denman Nov 29 '13 at 15:28
yes Kobi K's answer seems to work but I do not understand the issue with communicate. I found that removing the "-tt" makes the thing work ; this is a mistery for me. – Jerome WAGNER Nov 29 '13 at 20:50
For cases where ssh needs to be run with the -tt option from python, like for interactive sessions that need input from the stdin, even for remote scripts that need sudo pw, executing the ssh script via os.system() would overcome the issue. Related post: https://stackoverflow.com/questions/3692387/start-interactive-ssh-session-from-python-script – Snidhi Sofpro Feb 18 '21 at 07:09

python subprocess communicate freezes

2 Answers2

Linked