Recursive SFTP listdir in Python?

Question

I need to recursively list the content of a directory that contains a lot of subdirectories (more than 16,000).

I am currently using Paramiko's SFTP client, which doesn't offer any recursive listdir functionality. So I have to first run listdir on the parent folder, and then another listdir for each of the (many, many) subdirectories. It takes too long to run.

Is there any way to run the recursive listdir in a single SFTP call? I'm not limited to the Paramiko package, it's just the package that we're currently using.

I cannot use Paramiko's ssh.exec_command('ls -R <path>') due to local administrative restrictions (it would just return an error message).

I have an answer here which almost gives what you want. Maybe you can modify accordingly. https://stackoverflow.com/a/66781222/5675288 — scientific_explorer, Mar 24 '21 at 13:43

Martin Prikryl · Accepted Answer · 2020-12-03T12:00:25.857

So I have to first run listdir on the parent folder, and then another listdir for each of the (many, many) subdirectories.

That's the correct approach.

For an example of an implementation, see
Python pysftp get_r from Linux works fine on Linux but not on Windows

Is there any way to run the recursive listdir in a single SFTP call? I'm not limited to the Paramiko package, it's just the package that we're currently using.

No. That's not limitation of Paramiko. SFTP protocol itself does not have any way to retrieve recursive listing. Not that other similar protocols (like FTP) have such functionality commonly, so it's not really a limitation.

Recursive SFTP listdir in Python?

1 Answers1

Linked

Related