How to automate the sending of files over the network using python?

Question

Here's what I need to do:

I need to copy files over the network. The files to be copied is in the one machine and I need to send it to the remote machines. It should be automated and it should be made using python. I am quite familiar with os.popen and subprocess.Popen of python. I could use this to copy the files, BUT, the problem is once I have run the one-liner command (like the one shown below)

scp xxx@localhost:file1.txt yyy@]192.168.104.XXX:file2.txt

it will definitely ask for something like

Are you sure you want to connect (yes/no)?

Password :

And if im not mistaken., once I have sent this command (assuming that I code this in python)

conn.modules.os.popen("scp xxx@localhost:file1.txt yyy@]192.168.104.XXX:file2.txt")

and followed by this command

conn.modules.os.popen("yes")

The output (and I'm quite sure that it would give me errors) would be the different comparing it to the output if I manually type it in in the terminal.

Do you know how to code this in python? Or could you tell me something (a command etc.) that would solve my problem

Note: I am using RPyC to connect to other remote machines and all machines are running on CentOS

Answer 1

The right way to do it via Python is really using fabric as stated in a comment above.

Create a file called fabfile.py

#!/usr/bin/python

from fabric.api import run, env, sudo, put

env.user = 'user'
env.hosts = ['xxxxx.org',]

def copy():
    put('wong_8066.zip', '/home1/user/wong_8066.zip')

Then on your local machine, run fab copy and it will prompt for password.

yeukhon@yeukhon-P5E-VM-DO:~$ fab copy
[xxxxxx.org] Executing task 'copy'
[xxxxxx.org] Login password: 
[xxxxxx.org] put: wong_8066.zip -> /home1/user/wong_8066.zip

Done.

You can do a lot more with fabric. Fabric is used for deployment. You can use fabric to run things locally, or remotely to mutliple host. You can pass many options when u are running.

If you want to automated this without password prompt, you can specify the path to your ssh-key as stated in the documentation.

fab uses Python’s optparse library, meaning that it honors typical Linux or GNU style short and long options, as well as freely mixing options and arguments. E.g. fab task1 -H hostname task2 -i path/to/keyfile is just as valid as the more straightforward fab -H hostname -i path/to/keyfile task1 task2.

Answer 2

This question seems to cover what you are trying to do.

Use shutil.copyfile() to use the OS level copy utility

In case you don't want to click the link, here it is:

import shutil
source_path = r"\\mynetworkshare"
dest_path = r"C:\TEMP"
file_name = "\\myfile.txt"

shutil.copyfile(source_path + file_name, dest_path + file_name)

Answer 3

per my experience, use sftp the first time will prompt user to accept host public key, such as

The authenticity of host 'xxxx' can't be established. RSA key fingerprint is xxxx. Are you sure you want to continue connecting (yes/no)? once you input yes, the public key will be saved in ~/.ssh/known_hosts, and next time you will not get such prompt/alert.

To avoid this prompt/alert in batch script, you can use turn strict host check off use

scp -Bqpo StrictHostKeyChecking=no

but you are vulnerable to man-in-the-middle attack.

you can also choose to connect to target server manually and save host public key before deploy your batch script.