Python s3 invalid archive file

Question

I am trying to zip a file on s3 using the code below,, the issue is, file seems to be created but when i try to open the zip file it shows error that archive is not valid. any idea what I am doing wrong here...

code i am using is as below...

def upload_zip_file(event,context):    
    try:
        bucket_name ="s3_bucket"
        bucket_path ="files"
        s3 = boto3.client('s3')
        data_io = BytesIO()
        filename = "FILE.zip"
        funct_name ="FILE.json"
        zip_buffer = io.BytesIO()
        with zipfile.ZipFile(zip_buffer, "a", zipfile.ZIP_DEFLATED, False) as zipper:
            infile_object = s3.get_object(Bucket=bucket_name, Key=bucket_path +"/"+ funct_name) 
            infile_content = infile_object['Body'].read()
            zipper.writestr(filename, infile_content)
            s3.put_object(Bucket=bucket_name, Key=bucket_path +"/"+filename, Body=zip_buffer.getvalue())            
            
    except Exception:
        print(traceback.format_exc())

from reading the code you posted I'm inferring the processing is being done in an aws lambda function, is that correct? — Lucas Roberts, Nov 30 '21 at 17:42
I'm confused by what you mean when you say "when i try to open the zip file", from the string literal assigned to `funct_name`, it looks like the code reads a json file, not necessarily a zip file, i.e. the code ***writes*** to a zipfile. Is that the correct intent of the code? — Lucas Roberts, Nov 30 '21 at 18:01
Yes,, the intention is to read a json file and create a zip file from that json file and put it on s3 — Faisal Niazi, Nov 30 '21 at 18:26

kgiannakakis · Answer 1 · 2021-11-30T18:32:42.470

0

You need to move the put_object, outside of the with, so that the file handle is properly closed.

zip_buffer = io.BytesIO()
with zipfile.ZipFile(zip_buffer, "a", zipfile.ZIP_DEFLATED, False) as zipper:
    infile_object = s3.get_object(Bucket=bucket_name, Key=filename) 
    infile_content = infile_object['Body'].read()
    zipper.writestr(filename, infile_content)
    
s3.put_object(Bucket=bucket_name, Key='test.zip', Body=zip_buffer.getvalue())

edited Nov 30 '21 at 18:32

answered Nov 30 '21 at 17:52

kgiannakakis

103,016
27
158
194

thanks for the reply.....its a JSON file , and adding infile_content = infile_object['Body'].read().decode('utf-8') does not seem to work – Faisal Niazi Nov 30 '21 at 18:02
read() returns bytes. You need to decode it to use it as a string (https://stackoverflow.com/questions/31976273/open-s3-object-as-a-string-with-boto3). I was under the impression that writestr requires a string and not a bytes object. – kgiannakakis Nov 30 '21 at 18:05
@Faisal Niazi are you getting the same error message or something else? – Lucas Roberts Nov 30 '21 at 18:15
@LucasRoberts its the same error of invalid archive – Faisal Niazi Nov 30 '21 at 18:21
You need to close the file and then upload, see my edited answer. – kgiannakakis Nov 30 '21 at 18:35
The file object is actually in memory as an io.BytesIO() object, so is should be OK. – kgiannakakis Nov 30 '21 at 18:41
modified as per your suggestion,, but still same error.... :( – Faisal Niazi Nov 30 '21 at 19:21

Python s3 invalid archive file

1 Answers1