Multiple Uploads of unknown number of files to S3 with JetS3t

Question

from the sample code examples of jetS3t the code that follows: What to do if I don't know the number of files to be downloaded from the beginning? E.g an application with thumbnails in S3, and a client application that the user scrolls through a list of thumbnails.

Download objects to local files

The multi-threading services provide a method to download multiple objects at a time, but to use this you must first prepare somewhere to put the data associated with each object. The most obvious place to put this data is into a file, so let's go through an example of downloading object data into files. To download our objects into files we first must create a DownloadPackage class for each object. This class is a simple container which merely associates an object with a file, to which the object's data will be written. Create a DownloadPackage for each object, to associate the object with an output file.

 DownloadPackage[] downloadPackages = new DownloadPackage[5];
   downloadPackages[0] = new DownloadPackage(objects[0],
       new File(objects[0].getKey()));
   downloadPackages[1] = new DownloadPackage(objects[1],
       new File(objects[1].getKey()));
   downloadPackages[2] = new DownloadPackage(objects[2],
       new File(objects[2].getKey()));
   downloadPackages[3] = new DownloadPackage(objects[3],
       new File(objects[3].getKey()));
   downloadPackages[4] = new DownloadPackage(objects[4],
       new File(objects[4].getKey()));

   // Download the objects.
   simpleMulti.downloadObjects(bucket, downloadPackages);
   System.out.println("Downloaded objects to current working directory");

Any suggestions for that cases? Thanks in advance

Antonis

aviad · Answer 1 · 2011-05-23T21:35:59.993

You can list the objects in the bucket you download from and filter keys according to the rules you define. Then you can start a multipart download.

get keys from the bucket:



 public List getFilesList(String accessKey, String secretKey
            ,String bucketName ,String directoryPathRelativeToBucket) {
        List keys = new ArrayList();
        org.jets3t.service.model.S3Object[] objects = new org.jets3t.service.model.S3Object[]{};
        try {
            // Create a credentials object and service to access S3 account
            org.jets3t.service.security.AWSCredentials myCredentials =
                new org.jets3t.service.security.AWSCredentials(accessKey, secretKey);

        S3Service service = new RestS3Service(myCredentials);

        objects = service.listObjects(bucketName
                ,directoryPathRelativeToBucket, null);
        log.info("got bucket listing for bucket[" + bucketName + "]");
    } catch (S3ServiceException e) {
        log.error("Failed to get or object listing for bucket[" + bucketName + "] due to exception:", e);
    }

    for (org.jets3t.service.model.S3Object s3Object : objects) {
        if (s3Object.getKey().contains("$") == false) {
            keys.add(s3Object.getKey());
        }
    }
    return keys;
}

after filtering the keys you can download particular part of the key-list.

Multiple Uploads of unknown number of files to S3 with JetS3t

1 Answers1