0

Am using python and paramiko to copy a 5GB file between server A and Server B and script will be executed from serverX, which will open a ssh session to serverb from serverX and run the command to copy the file from server B using sshpass. Script is working, but it is not copying the complete 5GB file. it's copying only half and some time less than half.

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(serverb, username=user, password=password)

try:
    stdin, stdout, stderr = client.exec_command("sshpass -p password scp -v -r root@serverA:/tmp/file_to_copy_name /tmp/",timeout=None)
except Exception as err:
    print("copy between server error")
    raise
pynexj
  • 19,215
  • 5
  • 38
  • 56
SUN
  • 87
  • 1
  • 12

2 Answers2

1

You may want to use the Rsync over SSH instead of scp (secure remote file copy) with sshpass (noninteractive ssh password provider). It supports the fast incremental file transfer (can resume unfinished upload) and using the SSH key is much more secure than passing the raw password via sshpass.

Something like: 

rsync -az /root/bigfile.txt 198.211.117.129:/root/

-a for archive mode -z to compress file data during the transfer

The manual: https://download.samba.org/pub/rsync/rsync.html
Moreover, it can resume the copy started with scp.

Here is the instruction on how to use it over SSH: https://www.digitalocean.com/community/tutorials/how-to-copy-files-with-rsync-over-ssh

Also, as already pointed out by @pynexj, the client.exec_command() will not wait until the command execution will be finished. So you may want to have some alternative way to check if the file was successfully copied and have the same data as the source. One of the options could be checking the MD5 hash: https://stackoverflow.com/search?q=Python+md5+hash

And you may want to check the: What is the fastest hash algorithm to check if two files are equal?

AI Mechanic
  • 631
  • 7
  • 8
  • BTW, if you need to rsync via ssh with custom port: `rsync -rvz -e 'ssh -p 8882' --progress root@198.211.117.129:/root/bigfile.txt ./dir/` – AI Mechanic Jun 09 '21 at 20:50
1

I guess you can use

rsync -avP --partial source target

where source or target can be both the remote server path or the local server path in your required order.

aboger
  • 2,214
  • 6
  • 33
  • 47
DevilWhite
  • 51
  • 6