how to execute and monitor a long-running process remotely and view output before completion

Question

how can I execute a process on a remote machine and view the standard output before the process exits? ssh is ideal but only reports output once the remote command has completed.

an example python script like this:

#!/usr/bin/python3
import time

tStart = time.time()
elapsedTime=0

while elapsedTime<10000:
    time.sleep(2)
    elapsedTime = time.time() - tStart
    print('elapsed time: {0:6.2f}(s)'.format(elapsedTime))

generates output indicating it's progress:

elapsed time:   2.00(s)
elapsed time:   4.00(s)
elapsed time:   6.00(s)
elapsed time:   8.00(s)
...

I'd like to run this on a remote machine with ssh:

ssh vector /home/comperem/myProc.py

This process will not end for many minutes or hours and I need to monitor the process as it runs.

How can I get ssh or something similar to run the command remotely and report stdout to the local console as it is generated, before the remote process completes? Output after each line would be useful.

Try `tmux` or `screen`. You can also redirect output to system log by `logger`. — Łukasz Jakubek, Jul 20 '18 at 05:27
yes, I've changed my use-case to remotely start detached screen processes with parallel-ssh python library. but the basic functionality was solved below by Kenster — Marc Compere, Jul 20 '18 at 20:54

score 1 · Answer 1 · answered Jul 20 '18 at 15:50

ssh vector /home/comperem/myProc.py

When you invoke a remote program through ssh in this fashion, the remote process runs without a TTY by default. Instead, it is connected to the ssh server through a set of pipes. Your python program is probably buffering its output because it's writing to a pipe rather than a TTY.

The simplest fix is for your program to flush its output. This question discusses how to get a python script to flush standard output. I'm not a python programmer, but it looks like this should work:

while elapsedTime<10000:
    time.sleep(2)
    elapsedTime = time.time() - tStart
    print('elapsed time: {0:6.2f}(s)'.format(elapsedTime), flush=True)
                                                         ^^^^^^^^^^^^

The other question discusses other ways to do it.

outstanding. flushing the buffer was exactly what was needed. it may have worked in block-style output had I been patient enough to wait for enough the buffer to fill. nice solution! — Marc Compere, Jul 20 '18 at 20:51

score 0 · Answer 2 · answered Aug 04 '19 at 21:50

You can resolve your question with daggy:

Create Data Aggregation Config config.yaml like this:

aliases:
 - &myProc
   name: myProc
   command: tail -f /home/comperem/myProc.py
   extension: log
 - &auth_ssh
   login: user
   password: password

sources:
   host1:
     type: ssh
     connection:
       *auth_ssh
     host: 192.168.1.10
     commands:
       - *myProc
    hostN:
     type: ssh
     connection:
       *auth_ssh
     host: 192.168.1.N
     commands:
       - *myProc

Run daggy -o my_proc_folder config.yaml
Run tail -f my_proc_folder/*

how to execute and monitor a long-running process remotely and view output before completion

2 Answers2