Stack Overflow
Stack Overflow

Questions tagged [boto3]

Boto 3 - The Amazon Web Services (AWS) SDK for Python

About

Boto is the Amazon Web Services (AWS) software development kit (SDK) for Python, which allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2. Boto provides an easy to use, object-oriented API as well as low-level direct service access.

Links

7627 questions
408
votes
2 answers

Difference in boto3 between resource, client, and session?

I am using Python 2.7.12 in Ubuntu 16.04 LTS. I'm learning how to use boto3 from the following link: https://boto3.readthedocs.io/en/latest/guide/quickstart.html#using-boto-3. My doubt is when to use resource, client, or session, and their…
shiva
  • 5,083
  • 5
  • 23
  • 42
359
votes
21 answers

Listing contents of a bucket with boto3

How can I see what's inside a bucket in S3 with boto3? (i.e. do an "ls")? Doing the following: import boto3 s3 = boto3.resource('s3') my_bucket = s3.Bucket('some/path/') returns: s3.Bucket(name='some/path/') How do I see its contents?
Amelio Vazquez-Reina
  • 91,494
  • 132
  • 359
  • 564
342
votes
25 answers

check if a key exists in a bucket in s3 using boto3

I would like to know if a key exists in boto3. I can loop the bucket contents and check the key if it matches. But that seems longer and an overkill. Boto3 official docs explicitly state how to do this. May be I am missing the obvious. Can anybody…
Prabhakar Shanmugam
  • 5,634
  • 6
  • 25
  • 36
329
votes
13 answers

boto3 client NoRegionError: You must specify a region error only sometimes

I have a boto3 client : boto3.client('kms') But it happens on new machines, They open and close dynamically. if endpoint is None: if region_name is None: # Raise a more specific error message that will give #…
WebQube
  • 8,510
  • 12
  • 51
  • 93
311
votes
10 answers

How to handle errors with boto3?

I am trying to figure how to do proper error handling with boto3. I am trying to create an IAM user: def create_user(username, iam_conn): try: user = iam_conn.create_user(UserName=username) return user except Exception as e: …
SQDK
  • 3,987
  • 3
  • 18
  • 16
266
votes
5 answers

How to choose an AWS profile when using boto3 to connect to CloudFront

I am using the Boto 3 python library, and want to connect to AWS CloudFront. I need to specify the correct AWS Profile (AWS Credentials), but looking at the official documentation, I see no way to specify it. I am initializing the client using the…
Nader A. Jabbar
  • 2,856
  • 2
  • 12
  • 7
235
votes
8 answers

Open S3 object as a string with Boto3

I'm aware that with Boto 2 it's possible to open an S3 object as a string with: get_contents_as_string() Is there an equivalent function in boto3 ?
Gahl Levy
  • 3,211
  • 2
  • 13
  • 7
219
votes
17 answers

Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials

When I simply run the following code, I always gets this error. s3 = boto3.resource('s3') bucket_name = "python-sdk-sample-%s" % uuid.uuid4() print("Creating new bucket with name:", bucket_name) s3.create_bucket(Bucket=bucket_name) I have saved my…
d-_-b
  • 4,142
  • 6
  • 28
  • 43
213
votes
7 answers

How to specify credentials when connecting to boto3 S3?

On boto I used to specify my credentials when connecting to S3 in such a way: import boto from boto.s3.connection import Key, S3Connection S3 = S3Connection( settings.AWS_SERVER_PUBLIC_KEY, settings.AWS_SERVER_SECRET_KEY ) I could then use S3 to…
Robert Brax
  • 6,508
  • 12
  • 40
  • 69
206
votes
15 answers

Save Dataframe to csv directly to s3 Python

I have a pandas DataFrame that I want to upload to a new CSV file. The problem is that I don't want to save the file locally before transferring it to s3. Is there any method like to_csv for writing the dataframe to s3 directly? I am using…
user2494275
  • 2,063
  • 2
  • 13
  • 4
199
votes
8 answers

How to write a file or data to an S3 object using boto3

In boto 2, you can write to an S3 object using these methods: Key.set_contents_from_string() Key.set_contents_from_file() Key.set_contents_from_filename() Key.set_contents_from_stream() Is there a boto 3 equivalent? What is the boto3 method for…
jkdev
  • 11,360
  • 15
  • 54
  • 77
181
votes
7 answers

How to save S3 object to a file using boto3

I'm trying to do a "hello world" with new boto3 client for AWS. The use-case I have is fairly simple: get object from S3 and save it to the file. In boto 2.X I would do it like this: import boto key =…
Vor
  • 33,215
  • 43
  • 135
  • 193
175
votes
20 answers

Retrieving subfolders names in S3 bucket from boto3

Using boto3, I can access my AWS S3 bucket: s3 = boto3.resource('s3') bucket = s3.Bucket('my-bucket-name') Now, the bucket contains folder first-level, which itself contains several sub-folders named with a timestamp, for instance 1456753904534. I…
mar tin
  • 9,266
  • 23
  • 72
  • 97
168
votes
1 answer

What is the difference between the AWS boto and boto3

I'm trying to learn the boto API and I noticed that there are two major versions/packages for Python: boto and boto3. What is the difference between the AWS boto and boto3 libraries?
Matt
  • 3,592
  • 5
  • 21
  • 26
131
votes
10 answers

How to import a text file on AWS S3 into pandas without writing to disk

I have a text file saved on S3 which is a tab delimited table. I want to load it into pandas but cannot save it first because I am running on a heroku server. Here is what I have so far. import io import boto3 import os import pandas as…
alpalalpal
  • 1,583
  • 2
  • 12
  • 14
1
2 3
…
99 100