0

When using Paramiko to execute commands remotely, I can't see an updating progress bar using tqdm. I'm guessing this is because it isn't printing a new line when tqdm updates the bar

Here's a simple code example I've been using, but you'll need to supply your own SSH credentials

ssh = paramiko.SSHClient()

ssh.load_system_host_keys()
ssh.connect('8.tcp.ngrok.io', username=username, get_pty=True)

command = 'python3 -c "import tqdm; import time; [time.sleep(1) for i in tqdm.tqdm(range(5))]"'

stdin, stdout, stderr = ssh_client.exec_command('sudo -S '+command)
stdin.write(password+'\n')
stdin.flush()

###new_method
for l, approach in line_buffered(stdout):
    if approach=='print':
        print( l)
    if approach=='overlay':
        print( l, end='\r')

ssh.close()

Is there a way I can print the tqdm bar as it updates?

Based on Martin Prikryl's suggestion, I tried to incorporate the solution from:
Paramiko with continuous stdout
And adapted the code to print regardless of a new line

def line_buffered(f):
    line_buf = ""
    while not f.channel.exit_status_ready():
        # f.read(1).decode("utf-8")
        line_buf += f.read(1).decode("utf-8", 'ignore') 
        if line_buf.endswith('\n'):
            yield line_buf, 'print'
            line_buf = ''
        # elif len(line_buf)>40:
        elif line_buf.endswith('\r'):
            yield line_buf, 'overlay'

This does successfully print the as the output as it is generated, and reprints on the tqdm line, but when I run this code I get the following output

100%|| 5/5 [00:05<00:00,  1.00s/it]1.00s/it]1.00s/it]1.00s/it]1.00s/it]?, ?it/s]

Not very pretty, and getting swamped by the iteration time. It doesn't seem to be printing the actual progress bar. any ideas?

Martin Prikryl
  • 188,800
  • 56
  • 490
  • 992
Novice
  • 855
  • 8
  • 17

1 Answers1

1

It probably because you are (correctly) using non-interactive session to automate your command execution.

Most decently designed commands do not print an output intended for an interactive human use, like progress, when executed non-interactively.

If you really want to see the progress display, try setting get_pty argument of SSHClient.exec_command to true.

stdin, stdout, stderr = ssh_client.exec_command('sudo -S '+command, get_pty=True)
Martin Prikryl
  • 188,800
  • 56
  • 490
  • 992
  • Thanks! When I add get_pty the progress display goes from the stderr to the stdout, but only displays when the loop is finished. As for wh I want it, I am training a model remotely and would like to be able to execute everything from python (why I'm using paramiko), and track the progress since it can take several hours to train a model – Novice Apr 14 '21 at 09:06
  • So with your code, you were also getting the progress, but only when it finished? Or were you not getting the progress at all? + To read incomplete lines continuously, see https://stackoverflow.com/q/25260088/850848 – Martin Prikryl Apr 14 '21 at 09:11
  • I was seeing the progress bar after it finished. I tried that solution already and it didn't work for me. I don't think tqdm output a /n every time it updates, and that is how the implementation you linked prints its updates. I was wondering if there was a different character posted by tqdm every time it updates the progress bar – Novice Apr 14 '21 at 11:33
  • And also for reference, my code above does continuously print the output of the command in real time, but only prints when there is a new line – Novice Apr 14 '21 at 11:43
  • *"don't think tqdm output a /n every time it updates, and that is how the implementation you linked prints its update"* – Of course, but I assumed it's obvious and that you will update the example to print any output immediatelly. – Martin Prikryl Apr 14 '21 at 15:48
  • No, adapting the example you linked didn't help print the progress bar. It still just printed in the end. And it prints a new line every time so it kind of ruins the point of tqdm. I've been investigating on my own to see if there is a better way to print stdout, since I think it is treated as a file object, but I don't have a solution yet – Novice Apr 16 '21 at 11:32
  • Found the problem about real time printing, my code example had a problem in it. the list comprehension didn't have for in it. But I still have the issue that it's simply printing new lines, and can't tell exactly when the progress bar is ready to print – Novice Apr 16 '21 at 13:55
  • Did you test your latest code with `get_pty=True`? – Martin Prikryl Apr 16 '21 at 14:24
  • Yea, I've started leaving it in all the time so that my outputs get printed to stdout. I think there are other ways to do that with tqdm but this seems to work – Novice Apr 16 '21 at 15:10
  • And did you try it without `get_pty`, but reading `stderr`? – Martin Prikryl Apr 16 '21 at 15:37
  • Yep, same result – Novice Apr 19 '21 at 07:31