AWS S3 file search using Java

Question

We are using a java class to dowload a file from AWS s3 bucket with the following code

inputStream = AWSFileUtil.getInputStream(
            AWSConnectionUtil.getS3Object(null),
            "cdn.generalsentiment.com", filePath);

AWSFileUtil is a class which check the credentials and gets the inputstream from S3bucket using the getInputStream method.The filePath is the file inside cdn.generalsentiment.com bucket.

We want to write a method which can just check whether the particular file exists or not in the AWS S3 bucket and returns a boolean or some other value.

Please suggest me a solution for this.

public static boolean isValidFile(AmazonS3 s3,
        String bucketName,
        String path) throws AmazonClientException {
    try {
        ObjectMetadata objectMetadata =  
s3.getObjectMetadata("cdn.generalsentiment.com", path);
    } catch (NotFoundException nfe) {
        nfe.printStackTrace();
    }

    return true;
}

If the file exists it returns true, else it throws NotFoundException, which i want to catch and return the "isValidFile" method result as false. Guys any other alternative for the method body or return type would be great.

The updated one

public static boolean doesFileExist(AmazonS3 s3,
        String bucketName,
        String path) throws AmazonClientException,
        AmazonServiceException {
    boolean isValidFile = true;
    try {
        ObjectMetadata objectMetadata = 
s3.getObjectMetadata("cdn.generalsentiment.com", path);

    } catch (NotFoundException nfe) {
        isValidFile = false;
    }
   catch (Exception exception) {
        exception.printStackTrace();
        isValidFile = false;
    }
    return isValidFile;
}

score 25 · Accepted Answer · edited May 23 '17 at 10:32

25

Daan's answer using GET Bucket (List Objects) (via the respective wrapper from the AWS for Java, see below) is the most efficient approach to get the desired information for many objects at once (+1), you'll need to post process the response accordingly of course.

This is done most easily via one of the respective methods of Class AmazonS3Client, e.g. listObjects(String bucketName):

AmazonS3 s3 = new AmazonS3Client(); // provide credentials, if need be
ObjectListing objectListing = s3.listObjects(new ListObjectsRequest()
        .withBucketName("cdn.generalsentiment.com");
for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {
    System.out.println(objectSummary.getKey());
}

Alternative

If you are only interested in a single object (file) at a time, using HEAD Object will be much more efficient, insofar you can deduce existence straight from the respective HTTP response code (see Error Responses for details), i.e. 404 Not Found for a response of NoSuchKey - The specified key does not exist.

Again, this is done most easily via Class AmazonS3Client, namely getObjectMetadata(String bucketName, String key), e.g.:

public static boolean isValidFile(AmazonS3 s3,
        String bucketName,
        String path) throws AmazonClientException, AmazonServiceException {
    boolean isValidFile = true;
    try {
        ObjectMetadata objectMetadata = s3.getObjectMetadata(bucketName, path);
    } catch (AmazonS3Exception s3e) {
        if (s3e.getStatusCode() == 404) {
        // i.e. 404: NoSuchKey - The specified key does not exist
            isValidFile = false;
        }
        else {
            throw s3e;    // rethrow all S3 exceptions other than 404   
        }
    }

    return isValidFile;
}

edited May 23 '17 at 10:32

Community

1
1

answered Feb 24 '12 at 11:03

Steffen Opel

63,899
11
192
211

1

Whoa, I never even knew you could get the HEAD Object separately from the actual object. That's definitely a more efficient approach to find out whether a single file exists in S3. +1 to you too! – Daan Feb 24 '12 at 11:21
Wow! this solves my worry.. Thank you guys a lot. Btw small exception handline case which i edited in my actual post, please walkme through to handle that exception – Sangram Anand Feb 25 '12 at 13:43
@SangramAnand: I've included and modified the function you provided to address your question. – Steffen Opel Feb 25 '12 at 15:37
Thanks a lot for you walkthrough Steffen... Actually the class is not handling the NotFoundException, its going to a hibernate transaction class and then throwing exception their. I made some changes in code to fix the issue... please suggest me if the code is good. – Sangram Anand Feb 25 '12 at 15:57
@SangramAnand: I've fixed the code - S3 is actually not using dedicated exception types, rather only a generic [AmazonS3Exception](http://docs.amazonwebservices.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/AmazonS3Exception.html); as I said, the code has been from the top of my head, sorry for being misleading - I've actually tested the code now and it works for me like so, good luck! – Steffen Opel Feb 25 '12 at 17:16
Are there any constants available in the SDK for error codes, specifically something like AmazonS3Client.NO_SUCH_KEY - just to make things a bit more readable. – RTF Oct 16 '14 at 16:53
1

@RTF - there are quite some generic ones, check out [Constant Field Values](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/constant-values.html) for example; also, you'll find several enumerations for things like the [EC2 instance types](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/ec2/model/InstanceType.html) etc. – Steffen Opel Oct 16 '14 at 21:13
You may also want to check for 403. I see 403 when file does not exist, it may be due to the way security permissions are set. – Jasper May 05 '15 at 09:12

score 2 · Answer 2 · answered Feb 24 '12 at 10:34

2

Use the GET Bucket S3 API:

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGET.html

and specify the full file name as a prefix.

answered Feb 24 '12 at 10:34

Daan

3,403
23
19

score 1 · Answer 3 · edited Jun 20 '20 at 09:12

1

this is simple way to find existing folder in bucket. above answer also true. folder name contain '/' at last , it return true .

note: mybucket/userProfileModule/abc.pdf, it my folder structrue

    boolean result1 = s3client.doesObjectExist("mybucket", "userProfileModule/");
    System.out.println(result);

edited Jun 20 '20 at 09:12

Community

1
1

answered Dec 17 '18 at 16:33

dheeraj kumar

419
4
6

AWS S3 file search using Java

3 Answers3

Alternative

Linked