How to copy from one bucket to another bucket in s3 of certain suffix

Question

I have 3 buckets 1.commonfolder 2.jsonfolder 3.csvfolder.

Common folder will be having both json and csv files
need to copy all csv files to csvfolder
need to copy all json files to json folder

Code is below to get all the files from commonfolder How to copy after that

import boto3

s3 = boto3.client('s3')
def lambda_handler(event, context):
    #List all the bucket names
    response = s3.list_buckets()
    for bucket in response['Buckets']:
        print (bucket)
        print(f'{bucket["Name"]}')
        #Get the files of particular bucket
        if bucket["Name"] == 'tests3json':
         
            resp = s3.list_objects_v2(Bucket='commonfolder')
            for obj in resp['Contents']:
                files = obj['Key']
                print(files)

            if(filename.split('.')[1].lower()=='json'):
                copyjson(bucket,filename)
                #copyjson(jsonfolder,filename)
            elif(filename.split('.')[1].lower()=='csv'):
                copycsv(bucket, filename)
                #copycsv(csvfolder,filename)

need to create a new function copyjson,copycsv to do this job
Need to copy from common-bucket to either csv-bucket or json-bucket depending on the file extension

Does this answer your question? [Move files between two AWS S3 buckets using boto3](https://stackoverflow.com/questions/30161700/move-files-between-two-aws-s3-buckets-using-boto3) — sushanth, Jun 26 '20 at 03:27
Your code is a bit confusing... You mention 3 buckets, but then your code runs on a bucket called `tests3json`, but then it lists objects in a bucket called `commonfolder`. Is your requirement to copy from `common-bucket` to either `csv-bucket` or `json-bucket` depending on the file extension? How will this Lambda function be triggered (eg manually, on a schedule, or when a file is uploaded to S3)? — John Rotenstein, Jun 26 '20 at 05:36
@JohnRotenstein, Is your requirement to copy from common-bucket to either csv-bucket or json-bucket depending on the file extension? yes — aysh, Jun 28 '20 at 06:20
How will this Lambda function be triggered (eg manually, on a schedule, or when a file is uploaded to S3)? Should it only copy the file that caused the Lambda function to be triggered, or should it copy _all_ files in `common-bucket`? Please explain the workflow. — John Rotenstein, Jun 28 '20 at 07:14
It has to trigger the lambda by ` Add trigger `, only when file is uploaded in s3 then lambda will triggered and copy the files from common-bucket to extension bucket — aysh, Jun 28 '20 at 13:32
Moving files from one directory to another without the use buckets. — Red, Jun 29 '20 at 03:38
Let me know if my answer is not what you're looking for, so I can do further research. — Red, Jun 29 '20 at 04:04

score 5 · Accepted Answer · answered Jun 30 '20 at 04:13

You can check the following code:

import boto3

s3 = boto3.resource('s3')

def lambda_handler(event, context):
    
    source_bucket = s3.Bucket('01-commonfolder-231')
    json_bucket = s3.Bucket('02-jsonfolder-3435')
    csv_bucket = s3.Bucket('03-csvfolder-4552')
    
    for object in source_bucket.objects.all():
        
        #print(object)
        
        if object.key.endswith('.json'):
            
            print(f"{object.key} to json bucket")
      
            copy_object = json_bucket.Object(object.key)
            copy_object.copy({'Bucket': object.bucket_name,
                              'Key': object.key})                             
            
        elif object.key.endswith('.csv'):
            
            print(f"{object.key} to csv bucket")            

            copy_object = csv_bucket.Object(object.key)
            copy_object.copy({'Bucket': object.bucket_name,
                              'Key': object.key})

I tested this using my own sample buckets with test files:

aaa.json to json bucket
bbbbb.csv to csv bucket
bbbbb.json to json bucket
hhhh.csv to csv bucket

score 2 · Answer 2 · answered Jun 29 '20 at 03:52

You can use the move() method from shutil:

from shutil import move
from glob import glob

common_folder = 'C:\\Users\\User\\Desktop\\commonfolder\\'
csv_folder = 'C:\\Users\\User\\Desktop\\csvfolder\\'
json_folder = 'C:\\Users\\User\\Desktop\\jsonfolder\\'

for csv in glob(common_folder+"*.csv"):
    move(csv, csv_folder)

for json in glob(common_folder+"*.json"):
    move(json, json_folder)

How to copy from one bucket to another bucket in s3 of certain suffix

2 Answers2

Linked