How to loop through and iterate import of multiple csv files into AWS bucket?

Question

Boto3 provides upload process for single csv file...do you know how to iterate and loop through to upload multiple csv files from local directory to AWS bucket?

import boto3
s3 = boto3.resource('s3', aws_access_key_id='xx', aws_secret_access_key='yyy')
BUCKET = "bucketname"
s3.Bucket(BUCKET).upload_file("C:/Users/vrazerDesktop/Projects/Folder/filename.csv", "filename.csv")

I apologize but I'm new to coding. Tried this to aggregate csv files in a list, can't seem to find documentation to apply FOR loop for uploading the csv files from my list into the AWS S3 bucket

import os
import glob
import boto3
s3 = boto3.resource('s3', aws_access_key_id='xx', aws_secret_access_key='yyy')
BUCKET = "bucketname"
path = 'C:/Users/vRazer/Desktop/Projects/Folder'
extension = 'csv' os.chdir(path)
result = glob.glob('.{}'.format(extension))
list = [result]
for i in list:
    try:
        file = s3.Bucket(BUCKET).upload_file(list, '.{}'.format(extension))

File "", line 16 ^ SyntaxError: unexpected EOF while parsing –

What is the real question here? How to write a for loop? How to list CSV files in a folder? Something else? — jarmod, Mar 26 '20 at 20:05
How to write a for loop to upload csv files from local directory to AWS bucket? Either using Boto3 or any other method — VRazer, Mar 26 '20 at 21:13
I think you should take a stab at this. Write some code and update your question with that code and what problems you ran into. Find out how to list files using Python's `os.listdir()` or related. Yes, use boto3 to upload each file - there are many examples available to you. — jarmod, Mar 26 '20 at 21:42
import boto3 s3 = boto3.resource('s3', aws_access_key_id='xx', aws_secret_access_key='yyy') BUCKET = "bucketname" s3.Bucket(BUCKET).upload_file("C:/Users/vrazerDesktop/Projects/Folder/filename.csv", "filename.csv") — VRazer, Mar 26 '20 at 23:27
I wrote the above code for uploading single file, I wanted to know how to write the code with FOR Loop to upload multiple csv files from my folder into the AWS bucket. Please let me know if anyone else has done this type of upload before and what the code is...thanking in advance — VRazer, Mar 26 '20 at 23:29
Research the aforementioned `os.listdir()` function. This will give you a list of files. For example: https://stackoverflow.com/questions/9234560/find-all-csv-files-in-a-directory-using-python/38584736. Once you have a list of files, it's trivial to loop over that list and call your upload function. For example: https://www.geeksforgeeks.org/iterate-over-a-list-in-python/. Note that Stack Overflow isn't a code-writing service. It helps you fix code that you've written, not write it from scratch. — jarmod, Mar 26 '20 at 23:34
import os import glob import boto3 s3 = boto3.resource('s3', aws_access_key_id='xx', aws_secret_access_key='yyy') BUCKET = "cii-ds-dms" path = 'C:/Users/vRazer/Desktop/Projects/Folder' extension = 'csv' os.chdir(path) result = glob.glob('*.{}'.format(extension)) list = [result] for i in list: try: file = s3.Bucket(BUCKET).upload_file(list, '*.{}'.format(extension)) File "", line 16 ^ SyntaxError: unexpected EOF while parsing — VRazer, Mar 27 '20 at 00:32
Please don't write new code in the comments. You should add it to the question where everyone can easily see it, and where it is formatted as code for readability. — jarmod, Mar 27 '20 at 00:34
I tried the os.chdir() function. I'm getting to the point where I enumerate all the csv files in a list from my local firectory. However, I don't see documentation to apply FOR loop for Boto3 to upload the multiple csv files from my list to AWS S3 — VRazer, Mar 27 '20 at 00:35
OK, sounds like progress. You won't find anything in the boto3 documentation for how to do X in a for loop. Or in any other documentation. For loops are nothing to do with these libraries. They're a fundamental construct of the programming language. When you learn a programming language, you learn the syntax of the for loop (or its equivalents), and you learn how to use it. — jarmod, Mar 27 '20 at 00:42
Thank you for letting me know. I tried the syntax of for loop and seem to be getting an error because I don't understand how to apply the for loop inside the function to upload multiple files using the for loop. Or is my thinking that for loop can be applied to accomplish what I'm trying to do with Boto3 flawed? — VRazer, Mar 27 '20 at 00:49

jarmod · Accepted Answer · 2020-03-27T01:38:27.730

I've made a couple of minor modifications to the latest version of your code to upload all CSV in a folder. It will upload a file named friends.csv to csv-files/friends.csv, for example, but you can change that. I'm not using Windows so make sure that result is correct and contains a list of CSV file.

Hope this helps, and makes sense. Please feel free to ask about anything that's unclear.

import os
import glob
import boto3

s3 = boto3.resource('s3')
BUCKET = "bucketname"
path = 'C:/Users/vRazer/Desktop/Projects/Folder'
extension = 'csv'
os.chdir(path)
result = glob.glob('*.{}'.format(extension))

for file in result:
    try:
        print('Upload:', file)
        s3.Bucket(BUCKET).upload_file(file, 'csv-files/{}'.format(file))
    except Exception as e:
        print('Upload:', file, 'failed')
        print(e)

For Windows, I added r before the folder path, path = r' ' – VRazer Mar 27 '20 at 01:59 — VRazer, Mar 27 '20 at 01:59

How to loop through and iterate import of multiple csv files into AWS bucket?

1 Answers1