Python subprocess rsync using sshpass and specify port

Question

I've searched around for quite a bit, finding pieces of what I wish to achieve but not fully. I'm making a sync-script to synchronize files between two machines. The script itself is somewhat more advanced than this question (it provides possibility for both sides to request for file deletion and so on, no "master side").

First question

The following bash-command works for me:

rsync -rlvptghe 'sshpass -p <password> ssh -p <port>' <source> <destination>

how can I translate it into a python command to be used with the subprocess object?

I've managed to get the following python to work:

pw = getpass.getpass("Password for remote host: ")
command = ['sshpass', '-p', pw, 'rsync', '-rlvptgh', source, destination]
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while p.poll() is None:
   out = p.stdout.read(1)
   sys.stdout.write(out)
   sys.stdout.flush()

but it doesn't specify port (it uses standard 22, I want another one). To clarify, I wish to use similar code as this one but with the support for a specific port as well.

I have already tried to change the command to:

command = ['sshpass', '-p', pw, 'rsync', '-rlvptghe', 'ssh', '-p', '2222', source, destination]

which gives the following error:

ssh: illegal option -- r

and also many other variations such as for instance:

command = ['rsync', '-rlvptghe', 'sshpass', '-p', pw, 'ssh', '-p', '2222', source, destination]

Which gives the following error (where <source> is the remote host source host to sync from, ie variable source above command declaration):

Unexpected remote arg: <source>

How should I specify this command to nest them according to my first bash command?

Second question

When I've done all my searching I've found lots of frowning upon using a command containing the password for scp/rsync (ie ssh), which I use in my script. My reasoning is that I want to be prompted for a password when I do the synchronization. It is done manually since it gives feedback on filesystem modifications and other things. However, since I do 2 scp and 2 rsync calls I don't want to type the same password 4 times. That is why I use this approach and let python (the getpass module) collect the password one time and then use it for all the 4 logins.

If the script was planned for an automated setup I would of course use certificates instead, I would not save the password in clear text in a file.

Am I still reasoning the wrong way about this? Are there things I could do to strengthen the integrity of the password used? I've already realized that I should suppress errors coming from the subprocess module since it might display the command with the password.

Any light on the problem is highly appreciated!

EDIT:

I have updated question 1 with some more information as to what I'm after. I also corrected a minor copy + paste error in the python code.

Edit 2 explained further that I do have tried the exact same order as the first bash command. That was the first I tried. It doesn't work. The reason for changing the order was because it worked with another order (sshpass first) without specifying port.

Answer 1

I have found one way to solve this for my own needs. It includes invoking a shell to handle the command, which I avoided in the first place. It works for me though, but might not be satisfactory to others. It depends on the environment you want to run the command in. For me this is more or less an extension of the bash shell, I want to do some other things that are easier in python and at the same time run some bash commands (scp and rsync).

I'll wait for a while and if there's no better solution than this I will mark my answer as the answer.

A basic function for running rsync via python with password and port could be:

def syncFiles(pw, source, destination, port, excludeFile=None, dryRun=False, showProgress=False):
    command = 'rsync -rlvptghe \'sshpass -p ' + pw + ' ssh -p ' + port + '\' ' + source + ' ' + destination
    if excludeFile != None:
        command += ' --exclude-from='+excludeFile
    if dryRun:
        command += ' --dry-run'
    if showProgress:
        command += ' --progress'
    p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
    while p.poll() is None:
        out = p.stdout.read(1)
        sys.stdout.write(out)
        sys.stdout.flush()

The reason this works is as I wrote because the invoked bash shell handles the command instead. This way I can write the command exactly as I would directly in a shell. I still don't know how to do this without shell=True.

Note that the password is collected from the user with the getpass module:

pw = getpass.getpass("Password for current user on remote host: ")

It is not recommended to store your password in the python file or any other file. If you are looking for an automated solution it is better to use private keys. Answers for such solutions can be found by searching.

To call the scp-command with password the following python should do:

subprocess.check_output(['sshpass', '-p', pw, 'scp', '-P', port, source, destination])

I hope this can be useful to someone who wants to achieve what I am doing.

Answer 2

TLDR

use

sshpass -f /path/to/your/password/file

instead of -p option.

My attempt and questions below this line

I meet this problem in 2023, 9 years after you. The reason maybe come from your password

sshpass -p pw

I still have no ideal how python handle this pw var, but it seems a little bit different from bash shell.

I tried

sshpass -p cat /home/lee/.password

but this not work, And I am wondering why.

here is my code

#! /usr/bin/env python3.9
# -*- coding: utf8 -*-

import os
import shlex
from subprocess import Popen, PIPE, STDOUT


def process_sync(params_list):
    for params in params_list:
        cmd = (f"""/usr/bin/rsync -aP --timeout=15 --bwlimit={params['bwlimit']}""" 
               f""" -e "/usr/bin/sshpass -f {params['pwd_file_path']}  ssh -o StrictHostKeyChecking=no -l {params['user']} -p {params['port']}" """
               f""" {params['source_path']} {params['user']}@{params['host']}:{params['dest_path']}""")
        print(cmd)
        process = Popen(shlex.split(cmd), stdout=PIPE, stderr=STDOUT, bufsize=1, text=True, preexec_fn=os.setsid)
        try:
            for line in iter(process.stdout.readline, ''):
                print(line.strip())
        except KeyboardInterrupt:
            process.terminate()
            break


def main():
    params_list = [
        {
            'user': 'user',                                         # ssh username
            'port': '22',                                           # ssh port
            'host': '127.0.0.1',                                    # ssh host ip
            'pwd_file_path': '/path/to/your/pwd_file/path',         # ssh password path
            'source_path': '/path/to/your/source/path',
            'dest_path': '/path/to/your/dest/folder',
            'bwlimit': 200,                                         # bandwidth limit KB/s
        },
    ]
    process_sync(params_list)


if __name__ == '__main__':
    main()