S3ToSFTP: Move multiple files from same S3 key to SFTP path

Question

Requirement: Move multiple files from the same S3 Key to SFTP

Below is the part of the code, I was able to achieve moving one file into SFTP location. If the s3_key location has more than 1 file example as below, I need to get both files from /path/output to SFTP Location

/path/output/abc.csv
/path/output/def.csv

Tried: But both files are not posted

Tried passing s3_key as '/path/output/*.csv'

Code

with sftp.open(sftp_path + key_name, 'wb') as f:
     s3_client.download_fileobj(s3_bucket,s3_key, f)

Answer 1

See Boto3 to download all files from a S3 Bucket.

Except that instead of downloading the files to a local file:

client.download_file(bucket, k, dest_pathname)

... you will stream them to SFTP:

with sftp.open(sftp_path + file.get('Key'), 'wb', 32768) as f:
    client.download_file(bucket, k, f)

Note that this simple change does not handle "subfolders".

For the purpose of the 32768 argument, see Writing to a file on SFTP server opened using pysftp "open" method is slow.