1

In my S3 bucket, I have a folder structure of rootFolder1/year/month/day/myfile.csv. Depending on the time, the rootFolder folder could have multiple subfolder, organised by year, month and date, with each date folder containing multiple csv files. There could also be a rootFolder2 in my bucket, with the same folder structure.

I know for downloading object from S3, I can use the getObject method where I can pass in the params object with the bucket Key like so:

var params = {
        Key: "book/2020/01/01/all_books_000.csv",
        Bucket: 'my-bucket-test'
    }

    s3.getObject(params, (err, data) => {
        if (err) throw err;

        // do something with data
    })

But in my case I don't want to hardcode the Key in. Is there any elegant way I can call getObject where I can dynamically set the Key, so it downloads the files under each date folder?

edit: for some further clarity, these are the keys I get back when I call s3.listObjects

  'author/',
  'author/2020/',
  'author/2020/01/',
  'author/2020/01/01/',
  'author/2020/01/01/all_authors_000.csv',
  'book/',
  'book/2020/',
  'book/2020/01/',
  'book/2020/01/01/',
  'book/2020/01/01/all_books_000.csv',
  'book/2020/01/01/all_books_001.csv'
John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
blankface
  • 5,757
  • 19
  • 67
  • 114
  • You'll have to list the objects for a prefix as described [here](https://stackoverflow.com/questions/50366708/get-list-of-objects-located-under-a-specific-s3-folder/52063453) and then download those that you want individually. – Maurice Mar 22 '21 at 07:42
  • What do you mean by "downloads the files under each date folder"? Are you wanting to download _all_ objects in the bucket? Please clarify your requirements. – John Rotenstein Mar 22 '21 at 08:52
  • I mean downloading each of the `.csv` files – blankface Mar 22 '21 at 22:07

0 Answers0