Is it possible to transfer files from a directory using SCP in Python but ignore hidden files or sym links?

Question

I'm currently utilising Paramiko and SCPClient in Python to transfer a directory from one server to another as a means of backup. This works well however I do not want it to copy hidden files (.file_name) or symbolic links. Is this possible?

Unfortunately rsync isn't an option for me as it's not available on either of the remote servers I connect to. My script is below (sensitive info replaced with dummy data). Note I need to connect to a jump host before being able to connect to target_1 or target_2.

import os
import shutil
import time
import paramiko
from scp import SCPClient

#set up ssh variables
j_host = '00.00.00.00'
target_host_1 = '00.00.00.001'
target_host_2 = '00.00.00.002'
port_no = 22
username = ''
passw = ''

#set up temporary folder on local machine to store files
path = "/local_path/"
os.mkdir(path)

#create SSH Client for jump server
jump_host=paramiko.SSHClient()
jump_host.set_missing_host_key_policy(paramiko.AutoAddPolicy())
jump_host.connect(j_host, username=username, password=passw)

#set up channel to connect to 1 via jump server
jump_host_transport_1 = jump_host.get_transport()
src_addr = (j_host, port_no)
dest_addr_1 = (target_host_1, port_no)
jump_host_channel_1 = jump_host_transport_1.open_channel("direct-tcpip", dest_addr_1, src_addr)

#set up channel to connect to 2 via jump server
jump_host_transport_2 = jump_host.get_transport()
dest_addr_2 = (target_host_2, port_no)
jump_host_channel_2 = jump_host_transport_2.open_channel("direct-tcpip", dest_addr_2, src_addr)

#function which sets up target server, either 1 or 2
def create_SSHClient(server, port, user, password, sock):
    target=paramiko.SSHClient()
    target.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    target.connect(server, port, user, password, sock=sock)
    return target

#invoke above function to set up connections for 1 & 2
ssh_1 = create_SSHClient(target_host_1, port_no, username, passw, jump_host_channel_1)
ssh_2 = create_SSHClient(target_host_2, port_no, username, passw, jump_host_channel_2)

#delete old files in backup folder
command = "rm -rf /filepath/{*,.*}"
stdin, stdout, stderr = ssh_2.exec_command(command)
lines = stdout.readlines()
#print(lines)

#pause to ensure old directory is cleared
time.sleep(5)

#SCPCLient takes a paramiko transport as an argument, sets up file transfer connection
scp_1 = SCPClient(ssh_1.get_transport())
scp_2 = SCPClient(ssh_2.get_transport())

#get files from 1, store on local machine, put on 2
scp_1.get('/filepath/.', '/target_folder_local/', recursive=True)
scp_2.put('/target_folder_local/.', '/filepath/', recursive=True)

#remove temporary folder
shutil.rmtree(path)

#close connections
ssh_1.close()
ssh_2.close()
jump_host.close()

Answer 1

There's no API in SCPClient to skip hidden files or symbolic links.

---

For upload, it's easy, if you copy the SCPClient's code and modify it as you need. See the os.walk loop in _send_recursive function.

If you do not want to modify the SCPClient's code, you will have to iterate the files on your own, calling SCPClient.put for each. It will be somewhat less efficient, as it will start new SCP server for each file.

---

For download, you might be able to modify the SCPClient code to respond with non-zero code to C commands fed by the server for the files you do not want to download.

Check the _recv_file function. There where name is resolved, check for names or attributes of files you are not interested in downloading and do chan.send('\x01') and exit the function.

---

Though why do you want to use SCP? Use SFTP. It is much better suited for custom rules you need.

Paramiko does not have recursive SFTP transfer functionality (But pysftp does, see pysftp vs. Paramiko). But you won't be able to use it anyway, for the same reason you cannot use it with SCP. For your specific needs.

But check my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows. It shows a simple recursive SFTP download code. Just modify it slightly to skip the files you do not want to download.

Something like

if (not S_ISLNK(mode)) and (not entry.filename.startswith(".")):

(see Checking if a file on SFTP server is a symbolic link, and deleting the symbolic link, using Python Paramiko/pysftp)