saving from api to s3 bucket

Question

I'm trying to get the below python code to save the csv from an api to an amazon s3 bucket using bot03 and python, but I can't see where I'm going wrong. When I execute the code I don't get any error but the file never appear in the s3 bucket.

import boto3
from botocore.exceptions import ClientError

file_name = "test.csv"
bucket = "mybucket"

def main():
    url = "https://api0.solar.sheffield.ac.uk/pvlive/v3/pes/10?start=2021-01-01T00:00:00&end=2021-07-06T00:00:00&data_format=csv"
    x = requests.get(url,headers={'Content-Type': 'application/json', 'Accept': 'application/json', 'Accept-Encoding': 'gzip, deflate',})

    s3 = bot03.client("s3")
    with open("test.csv","rb") as file2:
     s3.upload_fileobj(x.content, bucket, "test.cvc")

any tips/advice would be appreciated. I'm a python/aws newbie so apologies if a basic question

score 1 · Answer 1 · answered Jul 12 '21 at 16:41

1

I used this code to acheive what I need

file_name = "test.csv"
bucket = "my_bucket"

def main():
    url = "https://api0.solar.sheffield.ac.uk/pvlive/v3/pes/10?start=2021-01-01T00:00:00&end=2021-07-06T00:00:00&data_format=csv"
    x = requests.get(url,headers={'Content-Type': 'application/json', 'Accept': 'application/json', 'Accept-Encoding': 'gzip, deflate',})



    s3_resource = boto3.resource('s3')
    s3_resource.Object(bucket, 'snowflake/csv/df1.csv').put(Body=x.content)


if __name__ == "__main__":
    main()

answered Jul 12 '21 at 16:41

WindSwept

77
1
2
13

Thank you for persisting with this and for putting the answer. You have saved me just in time for a presentation! – Sanchez333 Jun 24 '22 at 08:25
In case anyone else comes looking, the answer to this also has some useful code: https://stackoverflow.com/questions/40336918/how-to-write-a-file-or-data-to-an-s3-object-using-boto3 – Sanchez333 Jun 24 '22 at 08:57

score -1 · Answer 2 · answered Dec 01 '22 at 21:47

-1

per the OP(Original Post), did you try

(line 11) s3 = boto3.client("s3") -- OP: bot03.client("s3")

answered Dec 01 '22 at 21:47

Henry Wang

3
2

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Dec 04 '22 at 23:09
This does not provide an answer to the question. Once you have sufficient [reputation](https://stackoverflow.com/help/whats-reputation) you will be able to [comment on any post](https://stackoverflow.com/help/privileges/comment); instead, [provide answers that don't require clarification from the asker](https://meta.stackexchange.com/questions/214173/why-do-i-need-50-reputation-to-comment-what-can-i-do-instead). - [From Review](/review/late-answers/33317927) – naab Dec 07 '22 at 14:15

saving from api to s3 bucket

2 Answers2

Linked