1

Hi I am working on implementing a web based file explorer using the AWS S3 java SDK. One of the challenges that I am facing is implementing the folder navigation structure. We will probably have at most a few hundred folders that we need to retrieve from AWS but the only way I can see to get the folders is to go through every file in the bucket which will probably be 10s of thousands. This seems very wasteful. We were probably looking at having a task that runs every 30min or something to build a cache of directories but this is not an ideal solution. Any suggestions would be welcome.

EDIT: I Understand that S3 does not have a folder structure. What I am looking for is an efficient way of returning all of the prefixes without having to loop over all of the files. for example. If I have 3 files.

a/b/file.txt
a/b/file2.txt
b/file3.txt

I would like an efficient way of returning all of the unique prefixes. So I would get the following:
a/b
b

DarrenCibis
  • 865
  • 10
  • 25
  • S3 does not have the concept of folders. What you are calling "folders" are merely prefixes to the file "names". Related: http://stackoverflow.com/a/9330600/347777. Also, have a look at this doc from AWS: http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysHierarchy.html – Viccari Feb 12 '15 at 22:04
  • I have made an edit - I hope this makes what i am after more clear. I am not sure if there is a solution to my problem but I thought I would be worth asking. – DarrenCibis Feb 12 '15 at 22:47
  • Who decides what prefixes are created - is there some predictable structure, or are you handling arbitrary user-created filenames? – DNA Feb 12 '15 at 22:51
  • Its is user created. – DarrenCibis Feb 12 '15 at 23:00
  • I don't think there is an easy solution for this problem. S3 documentation about listing keys in a bucket usually refer to the link from my other comment (listing keys hierarchically) or the normal way of doing it (like you're doing). They also kind of suggest (here: http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysUsingAPIs.html) that you choose a common prefix for related objects. Since your key namespace is basically whatever customers choose, I don't see any way of doing it with S3 APIs without listing everything. – Viccari Feb 13 '15 at 17:08

0 Answers0