3

I need to download all content (including versions) of an Amazon S3 Bucket and upload in other Amazon S3 Bucket. Don't tell me to use aws, I just can't use.

I use tempfile.TemporaryFile for this, it apparently works, the print show that the file object has the right content inside, but the uploaded files are empty (zero bytes).

with tempfile.TemporaryFile() as data:
    sourceUser.download_fileobj('source-bucket',key,data)
    # next 2 lines was just to check the content of the file
    data.seek(0)
    print (data.read())
    destinationUser.upload_fileobj(data,'destination-bucket',key)
Claudiu
  • 577
  • 1
  • 9
  • 24
  • 2
    Re-seek to offset 0 immediately before the upload? Or open a new stream on the file? – jarmod Apr 12 '18 at 13:27
  • Have you seen [this question](https://stackoverflow.com/q/30161700/6779307)? – Patrick Haugh Apr 12 '18 at 13:34
  • @Patrick - I can't use the aws/aws cli/boto3 direct copy from one bucket to other. I just can't in my project :(. So I have to do the download and upload model :(. – Claudiu Apr 12 '18 at 17:54
  • 1
    @ClaudiuIvanescu you don't have to use any library to do a direct copy from one bucket to another. You can build and sign a request and send it to S3 using nothing but an hmac-sha-256 library and an HTTP user-agent... or a Lambda function behind API Gateway. – Michael - sqlbot Apr 13 '18 at 02:13
  • 1
    @Michael-sqlbot I'm sorry but I don't have enough knowledge for doing your way. jarmod showed the error and now works. But I don't want to take his credit! – Claudiu Apr 14 '18 at 16:59
  • 1
    This also sorted out a problem I was having, where I need to download and reupload (CV2 operations in my case need to happen). tmp.seek(0) also sorted my problem here. Thanks @jarmod – BizNuge Apr 28 '19 at 22:09

1 Answers1

1

I have the same requirement, How do I pass the NamedTemporaryFile to the upload s3

Not sure How to pass the NamedTemporaryFileName to output=f'{file_name}.gpg' and to the load_file function --> filename=f_source.name

with tempFile.NamedTemporaryFile("wb") as f_source:
                s3_client.download_fileobj(s3_bucket, s3_key, f_source)
                logger.info(f'{s3_key} file downloaded successfully to local {f_source}')
                f_source.flush()
                file_name = self.s3_key.split('/')[-1]
                gpg = gnupg.GPG()
                key_data = open(key_path).read()
                import_result = gpg.import_keys(key_data)
                f_source.seek(0)
                with open(f_source.name, 'r+b') as f:
                    status = gpg.encrypt_file(
                        file=f,
                        recipients=[recipient],
                        output=f'{file_name}.gpg',
                    )

                    s3_hook.load_file(
                        filename=f_source.name,
                        key=s3_key,
                        bucket_name=s3_bucket,
                        replace=True
                    )
Kar
  • 790
  • 13
  • 36
  • You don't have a name for temporary file.... you have the data content as f_source. Don't try to reopen. Do seek 0 and save as you like. – Claudiu Aug 01 '20 at 03:00