Paramiko channel stucks when reading large ouput

Question

I have a code where i am executing a command on remote Linux machine and reading the output using Paramiko. The code def looks like this:

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(IPAddress, username=user['username'], password=user['password'])


chan = self.ssh.get_transport().open_session()

chan.settimeout(10800)

try:
    # Execute thecommand
    chan.exec_command(cmd)

    contents = StringIO.StringIO()

    data = chan.recv(1024)

    # Capturing data from chan buffer.
    while data:
        contents.write(data)
        data = chan.recv(1024)

except socket.timeout:
    raise socket.timeout


output = contents.getvalue()

return output,chan.recv_stderr(600),chan.recv_exit_status()

The above code works for small outputs, but it gets stuck for larger outputs.

Is there any buffer related issue in here?

See also [Paramiko ssh die/hang with big output](https://stackoverflow.com/q/31625788/850848). — Martin Prikryl, Nov 25 '21 at 05:55

score 9 · Answer 1 · edited Jul 27 '16 at 15:31

9

I am posting the final code which worked with inputs from Bruce Wayne( :) )

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(IPAddress, username=user['username'], password=user['password'])

chan = self.ssh.get_transport().open_session()
chan.settimeout(10800)

try:
    # Execute the given command
    chan.exec_command(cmd)

    # To capture Data. Need to read the entire buffer to capture output
    contents = StringIO.StringIO()
    error = StringIO.StringIO()

    while not chan.exit_status_ready():
        if chan.recv_ready():
            data = chan.recv(1024)
            #print "Indside stdout"
            while data:
                contents.write(data)
                data = chan.recv(1024)

        if chan.recv_stderr_ready():            
            error_buff = chan.recv_stderr(1024)
            while error_buff:
                error.write(error_buff)
                error_buff = chan.recv_stderr(1024)

    exit_status = chan.recv_exit_status()

except socket.timeout:
    raise socket.timeout

output = contents.getvalue()
error_value = error.getvalue()

return output, error_value, exit_status

edited Jul 27 '16 at 15:31

Delimitry

2,987
4
30
39

answered Feb 12 '13 at 12:55

vipulb

545
2
4
10

This assumes that recv_ready() is true as soon as exit_status_ready() is true, right? I've seen cases where exit_status_ready() is true, but recv_ready() is not yet true, even though it has output eventually (i.e. if I manually block before reading it with 'while not stdout.channel.recv_ready(): pass'. The problem with that approach is that if there is NOT going to be any data, it gets stuck in an infinite wait loop. Is there a better way to do this? – David Doria May 07 '14 at 15:03
Yes i have encountered the same problem. So what i did was i am checking for chan.exit_status_ready() before the while loop. – vipulb May 12 '14 at 09:52
I don't understand - you just check exit_status_ready() once before the loop? How does that determine if recv_ready() is ever going to return? Can you please post a code snippet? – David Doria May 12 '14 at 12:57
ohh ok. I didn't try your scenario. I just reviewed my code. I actually never check for recv_ready(). I just check chan.exit_status_ready(), thats it. My code works fine. – vipulb May 19 '14 at 14:58
1

Be careful, `chan.exit_status_ready()` indicates that the remote proc finished but not that you finished reading the input buffers. That said, you might end up with no data if your code runs slow (or you just input a time.sleep(5) before the loop) as the channel signals that the remote proc finished and you just stopped reading. I'd suggest to check for: `chan.exit_status_ready() and not recv_ready() and not recv_stderr_ready()` for the loop condition followed by a chan.read() at the end to get any leftovers from the buffers. – tintin Sep 18 '15 at 20:17

bruce_w · Accepted Answer · 2013-02-11T14:00:24.390

i see no problem related to stdout channel, but i'm not sure about the way you are handling stderr. Can you confirm, its not the stderr capturing thats causing problem? I'll try out your code and let you know.

Update: when a command you execute gives lots of messages in STDERR, your code freezes. I'm not sure why, but recv_stderr(600) might be the reason. So capture error stream the same way you capture standard output. something like,

contents_err = StringIO.StringIO()

data_err = chan.recv_stderr(1024)
while data_err:
    contents_err.write(data_err)
    data_err = chan.recv_stderr(1024)

you may even first try and change recv_stderr(600) to recv_stderr(1024) or higher.

I ran the code for stdout. Wow it worked. But its not working for stderr. — vipulb, Feb 08 '13 at 11:33

score 2 · Answer 3 · answered Feb 13 '14 at 03:29

2

Actually I think all above answers can't resolve the real problem:

if the remote program produce large amount of stderr output first then

stdout.readlines()
stderr.readlines()

would hung forever. although

stderr.readlines()
stdout.readlines()

would resolve this case, but it will fail in case the remote program produce large amount of stdout output first.

I don't have a solution yet...

answered Feb 13 '14 at 03:29

fubupc

63
1
8

The outputs being produced first or second doesn't actually matter, these will still hang waiting for EOF if there are more than `2^21` characters total that get sent to the buffer you're not reading from first. I also don't have a great solution - you can use `stdout.channel.recv_ready()` to determine if there are any bytes in stdout, and then try to read just a certain amount of bytes, but I don't see an exposed method to determine how many bytes are currently in the buffer. You can also set a timeout with `stdout.channel.settimeout(int_time)` to prevent the channels from being blocking... – jeremysprofile Jan 07 '20 at 21:02
... but the downside in that case is that (I believe) the writing to the channel will raise exceptions about the buffer being full and you'll lose data. In my case, it was enough to realize that the order mattered, as none of the commands I'm doing will produce `2^21` characters to stderr. – jeremysprofile Jan 07 '20 at 21:06

jeremysprofile · Answer 4 · 2022-12-01T15:33:51.873

TL;DR: Call stdout.readlines() before stderr.readlines() if using ssh.exec_command() (assuming you will never write more than 2 billion characters to stderr)

If you use @Spencer Rathbun's answer:

sh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(IPAddress, username=user['username'], password=user['password'])

stdin, stdout, stderr = ssh.exec_command(cmd)

You might want to be aware of the limitations that can arise from having large outputs.

Experimentally, stdin, stdout, stderr = ssh.exec_command(cmd) will not be able to write the full output immediately to stdout and stderr. More specifically, a buffer appears to hold 2^21 (2,097,152) characters before filling up. If any buffer is full, exec_command will block on writing to that buffer, and will stay blocked until that buffer is emptied enough to continue. This means that if your stdout is too large, you'll hang on reading stderr, as you won't receive EOF in either buffer until it can write the full output.

The easy way around this is the one Spencer uses - get all the normal output via stdout.readlines() before trying to read stderr. This will only fail if you have more than 2^21 characters in stderr, which is an acceptable limitation in my use case.

I'm mainly posting this because I'm dumb and spent far, far too long trying to figure out how I broke my code, when the answer was that I was reading from stderr before stdout and my stdout was too big to fit in the buffer.

After hours of research, finally I found you. Thank you for saving my day! — Timmy Lin, Dec 01 '22 at 09:47

score 1 · Answer 5 · answered Feb 11 '13 at 15:18

1

It's easier if you use the high level representation of an open ssh session. Since you already use ssh-client to open your channel, you can just run your command from there, and avoid the extra work.

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(IPAddress, username=user['username'], password=user['password'])

stdin, stdout, stderr = ssh.exec_command(cmd)
for line in stdout.readlines():
    print line
for line in stderr.readlines():
    print line

You will need to come back and read from these files handles again if you receive additional data afterwards.

answered Feb 11 '13 at 15:18

Spencer Rathbun

14,510
6
54
73

Actually i need to use Transport Session cause i need to check the return code of cmds. – vipulb Feb 12 '13 at 12:46
@SpencerRathbun Do you not have to check stdout.channel.recv_ready() before using stdout.readlines()? It seems that sometimes readlines() returns nothing when there is definitely supposed to be something on stdout. – David Doria May 07 '14 at 15:01
@DavidDoria According to my read of the docs, we are getting a filelike object after the command ran and the streams have come back. We should not need to check, since everything is finished. If this is not correct, it's certainly a surprise to me. – Spencer Rathbun May 07 '14 at 17:03

d0n · Answer 6 · 2017-01-15T20:37:07.500

To have paramiko commands behave like a subprocess.call you may use this piece of code (tested with python-3.5 and paramiko-2.1.1):

#!/usr/bin/env /usr/bin/python3                                                

import os                                                                  
import sys                                                                                                                    
from paramiko import SSHClient, AutoAddPolicy               
from socket import getfqdn                                       

class SecureSHell(object):                                                 
    reuser = os.environ['USER']                                            
    remote = ''                                                            
    def __init__(self, *args, **kwargs):                                   
        for arg in args:                                                   
            if hasattr(self, arg):                                         
                setattr(self, arg, True)                                   
        for (key, val) in kwargs.items():                                  
            if hasattr(self, key):                                         
                setattr(self, key, val)

    @staticmethod                                                          
    def _ssh_(remote, reuser, port=22):                                    
        if '@' in remote:                                                  
            _reuser, remote = remote.split('@')                            
        _fqdn = getfqdn(remote)                                            
        remote = _fqdn if _fqdn else remote                                
        ssh = SSHClient()                                                  
        ssh.set_missing_host_key_policy(AutoAddPolicy()) 
        ssh.connect(remote, int(port), username=reuser)                                                                     
        return ssh                                                         

    def call(self, cmd, remote=None, reuser=None):                         
        remote = remote if remote else self.remote                         
        reuser = reuser if reuser else self.reuser              
        ssh = self._ssh_(remote, reuser)                                   
        chn = ssh.get_transport().open_session()                           
        chn.settimeout(10800)                                              
        chn.exec_command(cmd)                                              
        while not chn.exit_status_ready():                                 
            if chn.recv_ready():                                           
                och = chn.recv(1024)                                       
                while och:                                                 
                    sys.stdout.write(och.decode())                         
                    och = chn.recv(1024)                                   
            if chn.recv_stderr_ready():                                    
                ech = chn.recv_stderr(1024)                                
                while ech:                                                 
                    sys.stderr.write(och.decode())                         
                    ech = chn.recv_stderr(1024)                            
        return int(chn.recv_exit_status())                                 

ssh = SecureSHell(remote='example.com', user='d0n')                       
ssh.call('find')

Paramiko channel stucks when reading large ouput

6 Answers6

Linked