Python SFTP download files older than x and delete networked storage

Question

I'd like to download some files via sftp that are older than say 2 hours. Then I'd like to delete them from the network site. I can use the following code for sftp but handling objects on the remote machine is giving me problems. The code below fails at the 'timestamp = os.stat" line I believe it is an os module issue?

import paramiko, sys, os,time

host = 'ftp address'
port = 22
transport = paramiko.Transport((host, port))
password = "pass"                   #hard-coded
username = "user"                   #hard-coded
transport.connect(username = username, password = password)


sftp = paramiko.SFTPClient.from_transport(transport)
print 'SFTP Client initiated'

remotepath = "/remote folder/"
localpath = '/local folder/' 

for file in sftp.listdir('.'):
    fullpath   = os.path.join('.',file) 
    timestamp  = os.stat(fullpath).st_ctime # get timestamp of file
    createtime = datetime.datetime.fromtimestamp(timestamp)
    now = datetime.datetime.now()
    delta = now -createtime
    if delta.hours > 2:
        sftp.get(file,localpath) 
        sftp.remove(file)

sftp.close()
transport.close()

Martin Prikryl · Answer 1 · 2023-08-08T09:27:29.297

While the self-accepted answer by OP almost works, it's quite inefficient as it involves a roundtrip to the server for each file. While actually the code has all needed data already, it just throws it away by using pysftp.Connection.listdir wrapper, instead of using pysftp.Connection.listdir_attr directly.

for entry in sftp.listdir_attr(remotepath):
    timestamp = entry.st_mtime
    createtime = datetime.datetime.fromtimestamp(timestamp)
    now = datetime.datetime.now()
    delta = now - createtime
    if delta.hours > 2:
        filepath = remotepath + '/' + entry.filename
        sftp.get(filepath, os.path.join(localpath, entry.filename)) 
        sftp.remove(filepath)

Connection.listdir internally retrieves the same data as Connection.listdir_attr. They both ultimately call Paramiko SFTPClient.listdir_attr. But Connection.listdir returns only file names, discarding the all other file attributes. See also How to fetch sizes of all SFTP files in a directory through Paramiko.

Though note that pysftp seems dead. Consider using Paramiko directly instead. It has pretty much same API, so the above code will work as it is. See also pysftp vs. Paramiko.

Also, one should not use os.path.join on SFTP paths. SFTP uses always forward slashes, while os.path.join uses local path syntax, so on Windows, it would use backslashes and the code will fail.

And the target path of pysftp.Connection.get needs a file name too, not only a path (here the os.path.join should be used)

score 1 · Accepted Answer · answered Sep 20 '12 at 12:07

The bit required for getting the timestamp of the file on remote machine and then comparing it to now is below. Hacked together by a non-programmer (me) but it works.

timestamp  = sftp.stat(fullpath).st_atime  # get timestamp of file in epoch seconds
createtime = datetime.datetime.now()
now        = time.mktime(createtime.timetuple())
datetime.timedelta = now - timestamp

if datetime.timedelta> x:
    do something

Python SFTP download files older than x and delete networked storage

2 Answers2

Linked