0

I have a bucket on s3 containing a LOT of images (several 100 thousands). Each image has a very unique file name (a long hashed name) and are stored in a subdirectory named according to a corresponding object id in my database.

A simplified example:

/bucket/1/red-1.jpg
/bucket/1/red-2.jpg
/bucket/1/red-3.jpg
/bucket/2/blue-1.jpg
/bucket/2/blue-2.jpg
/bucket/2/blue-3.jpg

Now the problem is, that the id's in the database has been changed and by mistake the names of the subdirectories hasn't. This means that, according to my database, the image red-1.jpg should be in /bucket/2 but it actually still resides in /bucket/1. So I need to be able to search among all subdirectories, find the file called red-1.jpg, and move that to the right directory (in this case /bucket/2).

Note: The part about moving the files should be pretty straight forward, the biggest problem is by far how I can effectively search and find the path of those images, where I know the name, but not the current directory.

I'm on ruby, but I'm pretty open to using whatever.

PS. I saw something about amazons cloud search, but I don't know if that could be used for this..

Niels Kristian
  • 8,661
  • 11
  • 59
  • 117
  • Just as a heads up: there is no such thing as "folders" in S3. What you are calling folder is not more than a prefix to your filename. See: http://stackoverflow.com/questions/9329234/amazon-aws-ios-sdk-how-to-list-all-file-names-in-a-folder/9330600#9330600 – Viccari Dec 03 '12 at 18:09
  • Okay great, but how could this solve my problem, any idea? – Niels Kristian Dec 03 '12 at 18:12
  • I would try playing arround with the GET Bucket (List Objects) API call (http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGET.html). If needed, you can specify prefixes, but I don't think that will be your case. – Viccari Dec 03 '12 at 18:22

1 Answers1

1

There's no simple way to list only keys with a certain suffix.

You should iterate through all common prefixes (folders) with the list_objects method to find the object corresponding to the file name you are looking for.

tkotisis
  • 3,442
  • 25
  • 30