I have an Amazon S3 account with 1 million items in the root folder. Is there any Windows software that can be used to manage these files?
I've tried Bucket Explorer and CloudBerry, but both seize up trying to list this many files.
Asked
Active
Viewed 577 times
0
-
Did you consider using ssh, then a shell. And 1M files in a flat directory is a huge mistake. See http://stackoverflow.com/q/8735875/841108 – Basile Starynkevitch Jan 05 '12 at 17:25
-
@basile This is Amazon S3 I am using not a normal file system. You can't ssh, and the number of files in a directory is meaningless, it can be infinite. – soupagain Jan 10 '12 at 16:39
-
Yes the number of files can be 'infinite' S3 does not care, but basically all tools will care. I personally don't see how a GUI tool would work for you, unless there is one that you can set the delimiter and or marker directly. – Tom Andersen Jan 12 '12 at 14:56
-
If you are a programmer, you might use something like IronWorker and ruby to create tools that do what you want. – Tom Andersen Jan 12 '12 at 14:57
2 Answers
1
http://www.bucketexplorer.com/documentation/amazon-s3--search-on-objects-in-bucket.html Shows how to set delimiter and prefix up for a search. This might be what you need. Your files might have some set of common prefixes - file name prefixes, or a character thats common you can use as a delimiter to make folders - say there are a lot of numbers, then try a delimiter of '1' instead of the default '/'.
You may need to read up a bit to learn about delimiters and prefixes in S3.
Tom Andersen
- 7,132
- 3
- 38
- 55
-
Unfortunately, my files don't have identifiable prefixes or delimeters. – soupagain Feb 26 '12 at 09:33
-
If they are random, then pick a character, and you will get folders. – Tom Andersen Feb 27 '12 at 00:14
0
Well after trying many S3 tools I've finally found one which handles >million files with ease, and can do a sync as well. It's free, thought that wasn't important to me, I just wanted something that worked.
Dragon Disk:
soupagain
- 1,123
- 5
- 16
- 32