Get output continuously: rsync info=progress2 call in python script

Question

I'm calling rsync with popen and the output is not printing out continuously in my python script for my web application as it does in just normal linux. I'm trying to copy over all the files on one directory to another (massive copy). I want to use the progress numbers received from the output changing to eventually create/update a progress-bar I have in my web application. All I want is the total progress of the overall copy so I would use --info=progress2 in my rsync command. I also tried :

while True:
        line = self.proc.stdout.readline()
        if line != '':
            # the real code does filtering here
            print("test:", line.rstrip())
        else:
            break

but that waited till the end to just print test: b'' I think the issues is either with the while loop extracting data or how I print it to my console using a different class too.

There is not much information of using this --info=progress2 since it's a relatively new update.

Here's my code.

import subprocess
import logging
import sys
import os
import replicator.dfp.rep_utils as ru


class SyncProcessor(object):
    def __init__(self, src, dest):
        self.src = src
        self.dest = dest
        self.proc = None
        self.callback = None
        log_file = "sync-{}-{}.log".format(self.src, self.dest)
        self.sync_logger = ru.setup_logger(__file__, log_file, level=logging.DEBUG)

    def add_sync_callback(self, cb):
        self.callback = cb

    def run(self):
        print("Syncing Drive "+ str(self.src.driveNum) + " to Drive " + str(self.dest.driveNum))
        rsync_cmd = "sudo rsync -ah --info=progress2 --delete --stats /media/drive{}/ /media/drive{}".format(self.src.driveNum, self.dest.driveNum)
        self.proc = subprocess.Popen(rsync_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

        while self.proc.poll() is None:
            output = self.proc.stdout.readline()
            if output == '':
                break
            if output:
                print("OUTPUT DECODE: " + output.decode("utf-8")
                #self.sync_logger.info(output.decode())
                self.callback.update(output.decode())
        print("<< Finished Syncing >>")
        #self.sync_logger.debug("<< Finished Syncing >>")
        rc = self.proc.poll()
        #self.sync_logger.debug("Return code: {}".format(rc))
        os.system("sync")
        return rc

    def communicate(self):
        return self.proc.communicate()

class Progress(object):
    """Callback to report progress of a SyncProcessor"""
    def __init__(self, src, dest, out=sys.stdout):
        self.src = src
        self.dest = dest
        self.out = out

    def update(self, data):
        line = "From Progress({}-{}) -> {}"
    self.out.write(line.format(self.src, self.dest, data))

Possible duplicate of [read subprocess stdout line by line](https://stackoverflow.com/questions/2804543/read-subprocess-stdout-line-by-line) — gdlmx, Mar 07 '19 at 20:39

score 0 · Answer 1 · answered Mar 11 '19 at 17:11

So what I realized was the whole changing of percent from 0-100% was treated as one line since it was broken up by \r instead of \n

self.proc.stdout.readline()

therefore this line only activates after process reaches 100%

I switched it to self.proc.stdout.readline(80) which worked as it printed out every 80 characters giving me updates on the precentage. However since length of line changes throughout I'm looking for a better way to do this

Matt Lyon · Answer 2 · 2020-04-10T11:21:38.117

Had to do a lot of digging to find a working solution but the code below should print continuous updates. I found the solution in this thread.

import subprocess

run_list = ['rsync', '--info=progress2', '-a', 'src/', 'dest/']
with subprocess.Popen(
    run_list, stdout=subprocess.PIPE, bufsize=1, text=True
) as process:
    for line in iter(p.stdout.readline, b''):
        print(line.strip())

Additionally you can parse the line like so:

items = line.strip().split(' ')
print(tuple(filter(None, items))

Which will give you a tuple of bytes transferred, percentage complete, transfer speed and time remaining.

Note: in versions below Python3.7 you need to replace text=True with universal_newlines=True. text was introduced as an alias to universal_newlines in 3.7, see here for more detail.

score 0 · Answer 3 · answered Jun 10 '22 at 14:02

Tried this code from jisaacstone

import subprocess


process = subprocess.Popen([ 'rsync', '-avz',
                            '--info=progress2', source, destination],
                           stdout=subprocess.PIPE)

while process.poll() is None:
    line = process.stdout.readline()
    print(str(line))

I checked the output, formated it removed the b'' every "line" was the same length 46 and started with "/r" :

\r         32.768   0%    0,00kB/s    0:00:00  
\r  1.302.724.608  12%    1,21GB/s    0:00:07  
\r  2.930.114.560  27%    1,37GB/s    0:00:05  
\r  4.785.438.720  44%    1,49GB/s    0:00:03  
\r  5.572.165.632  51%    1,30GB/s    0:00:03  
\r  5.640.323.072  52%    1,01GB/s    0:00:04

so

process.stdout.readline(46)

did the trick for me

Hopfully someone will find this usefull

Get output continuously: rsync info=progress2 call in python script

3 Answers3