Read a JSON from S3

Question

I am new to python and following the answers from an existing_post I am trying to read a json file from amazon S3 like this:

  import boto3
  import os
    BUCKET = 'my_bucket'
    FILE_TO_READ = 'file.json'
    client = boto3.client('s3',
                           aws_access_key_id=os.environ.my_key,
                           aws_secret_access_key=os.environ.my_secret_key
                         )
    result = client.get_object(Bucket=BUCKET, Key='file') 
    text = result["Body"].read().decode()
    print(text['key']) # Use your desired JSON Key for your value

However I am receiving the following error:

NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

Can someone please tell me what am I doing wrong?

chris_b · Accepted Answer · 2020-07-30T12:07:33.073

The error is saying that that your key does not exist. I see that you already have the key "file.json" so I assume it exists. I would assume that you have your filed stored like this bucket/same_file/file.json . If this is the case try this:

import json
import boto3
s3_obj =boto3.client('s3')
s3_clientobj = s3_obj.get_object(Bucket='your_bucket', Key='file/file.json')
s3_clientdata = s3_clientobj['Body'].read().decode('utf-8')
print("printing s3_clientdata")
print(s3_clientdata)

If you want to do data manipualation, a more pythonic soution would be:

fs = s3fs.S3FileSystem()
with fs.open('yourbucket/file/your_json_file.json', 'rb') as f:
    s3_clientdata = json.load(f)
#Convert to df
df = pd.DataFrame(s3_clientdata,index=[0])

score 0 · Answer 2 · edited Jul 29 '20 at 12:25

0

I see you are retrieving the object with key "file" in

result = client.get_object(Bucket=BUCKET, Key='file')

I think you where meaning something like:

result = client.get_object(Bucket=BUCKET, Key=FILE_TO_READ)

See the get_object reference for further details

edited Jul 29 '20 at 12:25

deadshot

8,881
4
20
39

answered Jul 29 '20 at 12:24

Caliuf

63
6

Error says "The specified key does not exist". Try: Amazon console > S3 > Your bucket > Click on the file you want to download > Overview > Key. You should try using that key – Caliuf Jul 29 '20 at 12:51

Read a JSON from S3

2 Answers2