How to scp in Python?

Question

What's the most pythonic way to scp a file in Python? The only route I'm aware of is

os.system('scp "%s" "%s:%s"' % (localfile, remotehost, remotefile) )

which is a hack, and which doesn't work outside Linux-like systems, and which needs help from the Pexpect module to avoid password prompts unless you already have passwordless SSH set up to the remote host.

I'm aware of Twisted's conch, but I'd prefer to avoid implementing scp myself via low-level ssh modules.

I'm aware of paramiko, a Python module that supports SSH and SFTP; but it doesn't support SCP.

Background: I'm connecting to a router which doesn't support SFTP but does support SSH/SCP, so SFTP isn't an option.

EDIT: This is a duplicate of How to copy a file to a remote server in Python using SCP or SSH?. However, that question doesn't give an scp-specific answer that deals with keys from within Python. I'm hoping for a way to run code kind of like

import scp

client = scp.Client(host=host, user=user, keyfile=keyfile)
# or
client = scp.Client(host=host, user=user)
client.use_system_keys()
# or
client = scp.Client(host=host, user=user, password=password)

# and then
client.transfer('/etc/local/filename', '/etc/remote/filename')

score 159 · Answer 1 · edited Sep 04 '19 at 14:10

159

Try the Python scp module for Paramiko. It's very easy to use. See the following example:

import paramiko
from scp import SCPClient

def createSSHClient(server, port, user, password):
    client = paramiko.SSHClient()
    client.load_system_host_keys()
    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    client.connect(server, port, user, password)
    return client

ssh = createSSHClient(server, port, user, password)
scp = SCPClient(ssh.get_transport())

Then call scp.get() or scp.put() to do SCP operations.

(SCPClient code)

edited Sep 04 '19 at 14:10

Martin Prikryl

188,800
56
490
992

answered Nov 26 '10 at 03:03

Tom Shen

1,747
1
11
6

5

Except that module doesn't actually use paramiko - it's a lot of code that in the end for `scp` ends up actually calling the `scp` command line which only works on *nix. – Nick Bastin Jun 12 '12 at 06:49
8

Agreed, what you're seeing in the source code is the remote call to `scp -t` or `scp -f` ('to' and 'from' modes, which exist for this server-side purpose). This is essentially how scp works - it relies on another copy of scp existing on the server. This article explains it very well: https://blogs.oracle.com/janp/entry/how_the_scp_protocol_works – laher Oct 26 '13 at 05:29
10

This solution worked for me with two caveats: 1. these are the import statements I used: `import paramiko` `from scp import SCPClient` 2. Here is an example of the scp.get() command: `scp.get(r'/nfs_home/appers/xxxx/test2.txt', r'C:\Users\xxxx\Desktop\MR_Test')` – Chris Nielsen Jan 14 '16 at 15:49
This not only works great, but also takes advantage of SSH keys if they exist. – Marcel Wilson Jul 27 '16 at 19:28
Note that you can install this lib from PyPI as well: `pip install scp`, at version 0.10.2 as of this writing. – Andrew B. Jul 30 '16 at 16:05
what if you have a key file instead of a passphrase? – 3pitt Mar 03 '18 at 16:05
1

I'm getting an error `scp.SCPException: scp Protocol not available` . I'm trying to copy files from Windows to Mac. The same command works fine on the terminal. – Sankalp Jul 03 '19 at 10:45

score 17 · Answer 2 · edited Dec 30 '18 at 19:36

You might be interested in trying Pexpect (source code). This would allow you to deal with interactive prompts for your password.

Here's a snip of example usage (for ftp) from the main website:

# This connects to the openbsd ftp site and
# downloads the recursive directory listing.
import pexpect
child = pexpect.spawn ('ftp ftp.openbsd.org')
child.expect ('Name .*: ')
child.sendline ('anonymous')
child.expect ('Password:')
child.sendline ('noah@example.com')
child.expect ('ftp> ')
child.sendline ('cd pub')
child.expect('ftp> ')
child.sendline ('get ls-lR.gz')
child.expect('ftp> ')
child.sendline ('bye')

score 15 · Answer 3 · answered Jul 24 '16 at 20:14

Couldn't find a straight answer, and this "scp.Client" module doesn't exist. Instead, this suits me:

from paramiko import SSHClient
from scp import SCPClient

ssh = SSHClient()
ssh.load_system_host_keys()
ssh.connect('example.com')

with SCPClient(ssh.get_transport()) as scp:
   scp.put('test.txt', 'test2.txt')
   scp.get('test2.txt')

JimB · Answer 4 · 2011-03-07T02:11:56.833

11

You could also check out paramiko. There's no scp module (yet), but it fully supports sftp.

[EDIT] Sorry, missed the line where you mentioned paramiko. The following module is simply an implementation of the scp protocol for paramiko. If you don't want to use paramiko or conch (the only ssh implementations I know of for python), you could rework this to run over a regular ssh session using pipes.

scp.py for paramiko

edited Mar 07 '11 at 02:11

answered Oct 30 '08 at 20:22

JimB

104,193
13
262
255

Are you saying that the attached solution will securely transfer files to any machine running sshd, even if it's not running sftpd? That's exactly what I'm looking for, but I can't tell from your comment whether this just wraps sftp in an scp-like facade. – Michael Gundlach Dec 17 '08 at 19:45
2

This will transfer files with any machine running sshd, which has scp in the PATH (scp isn't part of the ssh spec, but it's fairly ubiquitous). This invokes "scp -t" on the server, and uses the scp protocol to transfer files, which has nothing to do with sftp. – JimB Dec 17 '08 at 20:14
1

Please note that I used openssh's implementation of scp as the model, so this isn't guaranteed to work with other versions. Some versions of sshd may also use the scp2 protocol, which is basically the same as sftp. – JimB Dec 17 '08 at 20:19
Could you please see this question: http://stackoverflow.com/questions/20111242/how-to-avoid-errno-12-cannot-allocate-memory-errors-caused-by-using-subprocess I am wondering if paramiko uses subprocess or comething else like sockets. I am having memory issues using subprocess.check_output('ssh blah@blah.com "cat /data/file*") due to clone/fork issues and am wondering if paramiko will have the same issues or not? – Paul Nov 25 '13 at 14:32
1

@Paul: paramiko will not have the same issues, since it does not use a subprocess. – JimB Nov 25 '13 at 19:37

score 10 · Answer 5 · edited May 12 '23 at 07:58

10

Simple example using paramiko's built in sftp client. That is not pure SCP only, but uses SSH below the surface as well, and should work whenever SCP works. As the OP states, his router prevents SFTP. So, if you absolutly need to use the SCP protocol, this solution is not for you.

import paramiko

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
 
client.connect('<IP Address>', username='<User Name>',password='' ,key_filename='<.PEM File path')
 
#Setup sftp connection and transmit this script 
print ("copying")

sftp = client.open_sftp() 
sftp.put(<Source>, <Destination>)


sftp.close()

Bonus using SFTP: you'll be able to list dirs for example, instead of being limited to put/get. Bonus 2, using only paramiko is reducing dependencies.

edited May 12 '23 at 07:58

benzkji

1,718
20
41

answered Jan 22 '20 at 08:36

shrikkanth roxor

237
1
3
8

3

It would help if you added some commentary to your answer explaining why your solution is a good one and help readers compare it with the other solutions available. Just posting code does not always help learner know how to recognise better solutions. – Brian Tompsett - 汤莱恩 Jan 22 '20 at 08:59
7

This is SFTP, not SCP. – Martin Prikryl Jan 22 '20 at 09:30
@MartinPrikryl yes, it is SFTP. But what's the point, both use SSH below the surface. And, with the SFTP, you can list dirs and the like, what you cannot with pure SCP. – benzkji May 11 '23 at 11:19
@benzkji The OP asked about SCP. So I assume OP need to use SCP specifically (the server might not support SFTP). So while *"use SFTP instead"* might be solution, the answer should at least acknowledge that. – Martin Prikryl May 11 '23 at 11:45
@MartinPrikryl I know. But, besides ultra akwards edge case configurations, SFTP should work as well, if SCP does. See also the archwiki: Since OpenSSH 9.0 the scp utility uses the SFTP protocol by default. The -O option must be used to use the legacy SCP protocol. And: Warning: The scp protocol is outdated, inflexible and not readily fixed https://wiki.archlinux.org/title/SCP_and_SFTP#Secure_copy_protocol_(SCP) – benzkji May 11 '23 at 11:51
1

@benzkji The OP explicitly says *"I'm connecting to a router which doesn't support SFTP"*. + It's not relevant that OpenSSH `scp` now uses SFTP under the hood. The question is about SCP protocol, not `scp` tool. And yes, SCP is outdated, so is this question. – Martin Prikryl May 11 '23 at 12:10

score 8 · Answer 6 · answered Oct 30 '08 at 14:42

8

if you install putty on win32 you get an pscp (putty scp).

so you can use the os.system hack on win32 too.

(and you can use the putty-agent for key-managment)

sorry it is only a hack (but you can wrap it in a python class)

answered Oct 30 '08 at 14:42

Blauohr

5,985
2
25
31

The only reason the 'nix one works is, you have SCP on the path; as Blauohr points out, that's not too hard to fix. +1 – ojrac Oct 30 '08 at 14:53

score 7 · Answer 7 · answered May 19 '18 at 08:40

7

As of today, the best solution is probably AsyncSSH

https://asyncssh.readthedocs.io/en/latest/#scp-client

async with asyncssh.connect('host.tld') as conn:
    await asyncssh.scp((conn, 'example.txt'), '.', recurse=True)

answered May 19 '18 at 08:40

Loïc

11,804
1
31
49

3

Hi. You make a claim about AsyncSSH being the best, but could you make a statement or 2 to support this claim? Maybe differentiate it from scp(), or how it mattered for your use case. (This said, it's vastly less code - in your example at least) – Scott Prive Aug 16 '19 at 18:18
3

@Crossfit_and_Beer hello. So I said "probably the best", I'll let the judgement to anyone :) I've not much time right now to compare AsyncSSH to other libraries. But very quickly : it's simple to use, it's asynchronous, it's maintained and the maintener is very nice and helpful, it is quite complete and very well documented. – Loïc Aug 17 '19 at 08:45
2

I went with this solution and it performs well, with over 100 simultaneous transfers. – Cow Feb 17 '21 at 12:18

score 6 · Answer 8 · answered Feb 07 '17 at 16:18

6

You can use the package subprocess and the command call to use the scp command from the shell.

from subprocess import call

cmd = "scp user1@host1:files user2@host2:files"
call(cmd.split(" "))

answered Feb 07 '17 at 16:18

user7529863

61
1
1

3

How is this substantially different from the base case mentioned by the questioner? Yes, subprocess is better than os.system, but in both cases, it doesn't work outside linux-like systems, has issues with password prompts, etc. – Foon Feb 07 '17 at 16:23

user443854 · Answer 9 · 2019-04-17T22:53:25.397

6

Have a look at fabric.transfer.

from fabric import Connection

with Connection(host="hostname", 
                user="admin", 
                connect_kwargs={"key_filename": "/home/myuser/.ssh/private.key"}
               ) as c:
    c.get('/foo/bar/file.txt', '/tmp/')

edited Apr 17 '19 at 22:53

answered Nov 23 '11 at 19:30

user443854

7,096
13
48
63

2

Could you be more specific? – Nick T May 08 '14 at 19:47
Fabric does not support scp as of now. (Reference: https://github.com/fabric/fabric/issues/1948) – akash Dec 29 '20 at 00:38

score 5 · Answer 10 · edited May 23 '18 at 18:16

5

It has been quite a while since this question was asked, and in the meantime, another library that can handle this has cropped up: You can use the copy function included in the Plumbum library:

import plumbum
r = plumbum.machines.SshMachine("example.net")
   # this will use your ssh config as `ssh` from shell
   # depending on your config, you might also need additional
   # params, eg: `user="username", keyfile=".ssh/some_key"`
fro = plumbum.local.path("some_file")
to = r.path("/path/to/destination/")
plumbum.path.utils.copy(fro, to)

edited May 23 '18 at 18:16

some bits flipped

2,592
4
27
42

answered Jul 05 '14 at 14:08

smheidrich

4,063
1
17
30

How can I enter the passphrase for the private key with Plumbum ? p – ekta Oct 20 '14 at 07:31
@ekta: You will be prompted for it on the command line. If you want to provide it from your own code, I don't think plumbum's API supports this. But plumbum's `SshMachine` does have access to `ssh-agent`, so if you just want to avoid typing it in every time, that's one way to do it. You could also file a feature request on plumbum's github page. – smheidrich Oct 20 '14 at 19:53

score 1 · Answer 11 · answered May 21 '12 at 13:03

1

I while ago I put together a python SCP copy script that depends on paramiko. It includes code to handle connections with a private key or SSH key agent with a fallback to password authentication.

http://code.activestate.com/recipes/576810-copy-files-over-ssh-using-paramiko/

answered May 21 '12 at 13:03

ccpizza

28,968
18
162
169

Thanks for the link and for taking time to publish good example of using paramiko. – Peter M. - stands for Monica May 21 '14 at 20:57

score 1 · Answer 12 · answered Jun 04 '14 at 23:26

1

If you are on *nix you can use sshpass

sshpass -p password scp -o User=username -o StrictHostKeyChecking=no src dst:/path

answered Jun 04 '14 at 23:26

user178047

1,264
15
20

The OP asked for a Pythonic solution – Ian May 26 '23 at 15:37

score 1 · Answer 13 · answered Oct 30 '08 at 16:20

Hmmm, perhaps another option would be to use something like sshfs (there an sshfs for Mac too). Once your router is mounted you can just copy the files outright. I'm not sure if that works for your particular application but it's a nice solution to keep handy.

How to scp in Python?

13 Answers13

Linked

Related