0

The following code works fine except if there is a subfolder, which does not have any file inside, then the subfolder will not appear in S3. e.g. if /home/temp/subfolder has no file, then subfolder will not show in S3. how to change the code so that the empty folder is also uploaded in S3? I tried to write sth. (see note below), but do not know how to call put_object() to the empty subfolder.

#!/usr/bin/env python
import os
from boto3.session import Session

path = "/home/temp"
session = Session(aws_access_key_id='XXX', aws_secret_access_key='XXX')
s3 = session.resource('s3')

for subdir, dirs, files in os.walk(path):
    # note: if not files ......
    for file in files:
        full_path = os.path.join(subdir, file)
        with open(full_path, 'rb') as data:

s3.Bucket('my_bucket').put_object(Key=full_path[len(path)+1:],    
Body=data)

besides, I tried to call this function to check if a subfolder or file exist or not. it works for file, but not subfolder. how to check if a subfolder exists or not? (if a subfolder exists, I will not upload)

def check_exist(s3, bucket, key):
    try:
        s3.Object(bucket, key).load()
    except botocore.exceptions.ClientError as e:
        return False
    return True

BTW, I refer the above code from

check if a key exists in a bucket in s3 using boto3

and

http://www.developerfiles.com/upload-files-to-s3-with-python-keeping-the-original-folder-structure/

thanks them for sharing the code.

user389955
  • 9,605
  • 14
  • 56
  • 98
  • This question is related to AWS S3 basic many times : S3 is an object store, all object name is actually a key name, it doesn't support folder. What you see using AWS console is just arbitrary. That's the reason why S3 using something call "PREFIX" to filter out object name that share similar prefix. This is the only way to let user organized and filter out object systematically like a folder. – mootmoot Nov 16 '17 at 18:39

1 Answers1

9

Directories (folders, subfolders, etc.) do not exist in S3.

When you copy this file to an empty S3 bucket /mydir/myfile.txt, only the file myfile.txt is copied to S3. The directory mydir is not created as that string is part of the file name mydir/myfile.txt. The actual file name is the full path, no subdirectories exist or are created.

S3 simulates directories by using a prefix when listing files in the bucket. If you specify mydir/, then all of the S3 objects that start with mydir/ will be returned including objects such as mydir/anotherfolder/myotherfile.txt. S3 supports a delimitor such as / so that the appearance of subdirectories can be created.

Note: There is no / at the beginning of a file name for S3 objects.

Listing Keys Hierarchically Using a Prefix and Delimiter

John Hanley
  • 74,467
  • 6
  • 95
  • 159
  • Thanks John. So there is no way to upload an empty subfolder? I just want to upload a whole directory including its structure to S3 for backup purpose. it is strange S3 does not have such kind of backup mechanism. It seems I have to compress the whole directory locally and then upload the single compressed file to S3. – user389955 Nov 15 '17 at 06:26
  • Directories don't exist on S3 so you cannot upload an empty directory. Note: some software products create simulated directory entries in S3 (CloudBerry). These are actually just 0 length files. If you want to use S3 as a backup clone of your local file system, you will run into issues as S3 is not a file system but a key value store. Creating a zip file and then uploading the zip file is a workable method. – John Hanley Nov 15 '17 at 06:31
  • 2
    @user389955 S3 is not hierarchical. Files are not "in" folders -- that's part of the reason S3 scales to essentially an infinite number of objects with no change in performance. You can upload an empty object with its name ending with `/` if you really want to be able to "find" a folder in S3, but it is just an illusion and tells you nothing about whether there's anything "in" it (and deleting it via the API will not delete the files "in" it). – Michael - sqlbot Nov 15 '17 at 06:36
  • Could such a thing happen if you were to use EFS? Or is that something completely different – Mangohero1 Nov 16 '17 at 18:55
  • Is you question, can you have empty folders in EFS? The answer is Yes. EFS is a file system whereas S3 is a flat key / value storage device. Think of S3 as having only one directory (the root directory). All files, not matter what their path name are called, are stored in the root directory. – John Hanley Nov 16 '17 at 19:16