How to append to json file on AWS S3

Question

I need to append to a json file on aws S3, python code is running on an EC2 instance.

In a local setting I can easily do this as follows:

import json

#example data
json_data = {"id": "123", "name": "XYZ", "transaction": [20.0, 30.0]}
# path
local_path = '/home/ubuntu/test.json'

with open(local_path, 'a', encoding='utf-8-sig') as file:
    json.dump(json_data, file)
    file.write('\n')
    file.close()

On EC2 I can connect to S3 as follows:

import boto
s3_open = boto.connect_s3(host='s3.eu-central-1.amazonaws.com')

I define the path to S3:

s3_path = 's3://my-bucket/test.json'

How can I append to this file using the logic described above?

When you say append you mean add to the remote file? or add to the local file then upload? S3 does not have an append action. — JBirdVegas, Jan 16 '20 at 16:42
Add to the remote file that is hosted on S3. What would be the alternative if S3 does not have an append action? — kanimbla, Jan 16 '20 at 16:43
Related to https://stackoverflow.com/questions/41783903/append-data-to-an-s3-object (quite similar question but language agnostic) — Yves M., Oct 19 '22 at 14:53

score 5 · Accepted Answer · answered Jan 16 '20 at 16:53

S3 does not have an append action what you can do is download the file, append it yourself then upload it as a new version of the same object.

Something along these lines.

import json
import boto3

s3_path = '/some/s3/path'
bucket = 'somebucket'
local_data = {"id": "123", "name": "XYZ", "transaction": [20.0, 30.0]}

s3 = boto3.client('s3')
resp=s3.get_object(Bucket=bucket, Key=s3_path)
data=resp.get('Body')

json_data = json.loads(data)
json_data.append(local_data)
s3.put_object(Bucket=bucket, Key=s3_path, Body=json.dumps(json_data).encode())

How to append to json file on AWS S3

1 Answers1