-1

My question is relevant to copy files from one AWS s3 bucket/folder to another AWS/S3 folder and also keep the deepest sub-folder name by pythons on databricks.

Is it possible to have a user upload from their local a folder structure containing all the folders/files temporarily, modify the object key names/paths (move, copy, delete?), then permanently upload to a S3 bucket? I'm mainly looking for better resources to read and learn.

I'm developing a web application, with 3rd party using Python, where a user can upload a single file OR single folder of files OR single folder of folders/files. Then they can place these uploads into user defined categories/subcategories via metadata.

Regarding single file(s), I understand AWS will create a folder in the object path. That main root 'folder' name will need to be changed for all three upload scenarios.

Single File(s) Example:

Folder Name A/File A,B,C

-Rename Folder Name A.

Regarding single folder of folders/files, the main root folder needs to be eliminated leaving 1st level folders as individual single folders moving into the permanent S3. This is a bulk option to upload many folders at once versus 'single folder'. The 'folder structure' downstream of the 1st level folders needs to stay intact and does not need to be altered.

Upload single folder of folders/files example:

Local Root Folder/Local Folder Name A/.../File A,B,C
Local Root Folder/Local Folder Name 1/.../File 1,2,3
Local Root Folder/Local Folder Name X/.../File X,Y,Z

-Eliminate *Local Root Folder* and rename *Local Folder Names A & 1 & X*

Any assistance with documentation and AWS features that could help explain the fundamentals behind this challenge would be much appreciated! Thanks!

RobinHood
  • 23
  • 5
  • Yes, you can just change the "folder" name in your application and they will be renamed - you just need to craft the object key how you like it and the structure will fall into place. – Ermiya Eskandary Apr 02 '22 at 13:25
  • I'm likely missing something really simple...I've read a lot that you cannot rename the folder/file name in S3 but you have to copy/delete to change. https://devopsmania.com/operation/how-to-rename-files-and-folders-in-amazon-s3/ – RobinHood Apr 02 '22 at 21:40
  • Correct yes - but you can change the object key before you upload the file. You can rename it as in changing the string in your Python application. – Ermiya Eskandary Apr 02 '22 at 21:52

1 Answers1

0

Objects in Amazon S3 cannot be 'renamed' or 'moved'.

Instead, you would need to copy the objects to a new Key (which can have a different path) and then delete the existing object.

The AWS Command-Line Interface (CLI) aws s3 mv command can be used to 'rename' a single object -- it will perform the Copy and Delete for you. However, this command cannot be used to 'rename' a folder. You would need to perform two operations:

  • aws s3 cp --recursive s3://source/folder s3://destination/folder
  • aws s3 rm s3://source/folder

My testing shows that aws s3 rm can remove a folder and its entire contents.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • Thanks John! Can you add (or subtract) multiple '/' layers to create a pseudo folder structure that contains more folders in the destination than the source? Example: S3://source/folder to s3://destination/another folder/another folder/folder – RobinHood Apr 03 '22 at 00:02
  • Yes, the `--recursive` option will copy all subfolders to the stated destination, eg `aws s3 cp --recursive s3://bucket/folder1 s3://buckert/folder2/folder3/folder4` – John Rotenstein Apr 03 '22 at 00:14