How to check if a specified key exists in a given S3 bucket using Java

Question

I would like to check if a key exists in a given bucket using Java. I looked at the API but there aren't any methods that are useful. I tried to use getObject but it threw an exception.

Answer 1

There's now a doesObjectExist method in the official Java API.

Enjoy!

Answer 2

Update:

It seems there's a new API to check just that. See another answer in this page: https://stackoverflow.com/a/36653034/435605

Original post:

Use errorCode.equals("NoSuchKey")

try {
    AmazonS3 s3 = new AmazonS3Client(new ClasspathPropertiesFileCredentialsProvider());
    String bucketName = getBucketName();
    s3.createBucket(bucketName);
    S3Object object = s3.getObject(bucketName, getKey());
} catch (AmazonServiceException e) {
    String errorCode = e.getErrorCode();
    if (!errorCode.equals("NoSuchKey")) {
        throw e;
    }
    Logger.getLogger(getClass()).debug("No such key!!!", e);
}

Note about the exception: I know exceptions should not be used for flow control. The problem is that Amazon didn't provide any api to check this flow - just documentation about the exception.

Answer 3

The right way to do it in SDK V2, without the overload of actually getting the object, is to use S3Client.headObject. Officially backed by AWS Change Log.

Example code:

public boolean exists(String bucket, String key) {
    try {
        HeadObjectResponse headResponse = client
                .headObject(HeadObjectRequest.builder().bucket(bucket).key(key).build());
        return true;
    } catch (NoSuchKeyException e) {
        return false;
    }
}

Answer 4

Using the AWS SDK use the getObjectMetadata method. The method will throw an AmazonServiceException if the key doesn't exist.

private AmazonS3 s3;
...
public boolean exists(String path, String name) {
    try {
        s3.getObjectMetadata(bucket, getS3Path(path) + name); 
    } catch(AmazonServiceException e) {
        return false;
    }
    return true;
}

Answer 5

In Amazon Java SDK 1.10+, you can use getStatusCode() to get the status code of the HTTP response, which will be 404 if the object does not exist.

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.AmazonS3Exception;
import org.apache.http.HttpStatus;

try {
    AmazonS3 s3 = new AmazonS3Client();
    ObjectMetadata object = s3.getObjectMetadata("my-bucket", "my-client");
} catch (AmazonS3Exception e) {
    if (e.getStatusCode() == HttpStatus.SC_NOT_FOUND) {
        // bucket/key does not exist 
    } else {
        throw e;
    }
}

getObjectMetadata() consumes fewer resources, and the response doesn't need to be closed like getObject().

---

In previous versions, you can use getErrorCode() and check for the appropriate string (depends on the version).

Answer 6

Use ListObjectsRequest setting Prefix as your key.

.NET code:

 public bool Exists(string key)
    {

        using (Amazon.S3.AmazonS3Client client = (Amazon.S3.AmazonS3Client)Amazon.AWSClientFactory.CreateAmazonS3Client(m_accessKey, m_accessSecret))
        {
            ListObjectsRequest request = new ListObjectsRequest();
            request.BucketName = m_bucketName;
            request.Prefix = key;
            using (ListObjectsResponse response = client.ListObjects(request))
            {

                foreach (S3Object o in response.S3Objects)
                {
                    if( o.Key == key )
                        return true;
                }
                return false;
            }
        }
    }.

Answer 7

For PHP (I know the question is Java, but Google brought me here), you can use stream wrappers and file_exists

$bucket = "MyBucket";
$key = "MyKey";
$s3 = Aws\S3\S3Client->factory([...]);
$s3->registerStreamWrapper();
$keyExists = file_exists("s3://$bucket/$key");

Answer 8

This java code checks if the key (file) exists in s3 bucket.

public static boolean isExistS3(String accessKey, String secretKey, String bucketName, String file) {

    // Amazon-s3 credentials
    AWSCredentials myCredentials = new BasicAWSCredentials(accessKey, secretKey); 
    AmazonS3Client s3Client = new AmazonS3Client(myCredentials); 

    ObjectListing objects = s3Client.listObjects(new ListObjectsRequest().withBucketName(bucketName).withPrefix(file));

    for (S3ObjectSummary objectSummary: objects.getObjectSummaries()) {
        if (objectSummary.getKey().equals(file)) {
            return true;
        }
    }
    return false;
}

Answer 9

Break your path into bucket and object. Testing the bucket using the method doesBucketExist, Testing the object using the size of the listing (0 in case not exist). So this code will do:

String bucket = ...;
String objectInBucket = ...;
AmazonS3 s3 = new AmazonS3Client(...);
return s3.doesBucketExist(bucket) 
       && !s3.listObjects(bucket, objectInBucket).getObjectSummaries().isEmpty();

Answer 10

The other answers are for AWS SDK v1. Here is a method for AWS SDK v2 (currently 2.3.9).

Note that getObjectMetadata and doesObjectExist methods are not currently in the v2 SDK! So those are no longer options. We are forced to use either getObject or listObjects.

listObjects calls are currently 12.5 times more expensive to make than getObject. But AWS also charges for any data downloaded, which raises the price of getObject if the file exists. As long as the file is very unlikely to exist (for example, you have generated a new UUID key randomly and just need to double-check that it isn't taken) then calling getObject is significantly cheaper by my calculation.

Just to be on the safe side though, I added a range() specification to ask AWS to only send a few bytes of the file. As far as I know the SDK will always respect this and not charge you for downloading the whole file. But I haven't verified that so rely on that behavior at your own risk! (Also, I'm not sure what how range behaves if the S3 object is 0 bytes long.)

    private boolean sanityCheckNewS3Key(String bucket, String key) {

        ResponseInputStream<GetObjectResponse> resp = null;
        try {
            resp = s3client.getObject(GetObjectRequest.builder()
                .bucket(bucket)
                .key(key)
                .range("bytes=0-3")
                .build());
        }
        catch (NoSuchKeyException e) {
            return false;
        }
        catch (AwsServiceException se) {
            throw se;
        }
        finally {
            if (resp != null) {
                try {
                    resp.close();
                } catch (IOException e) {
                    log.warn("Exception while attempting to close S3 input stream", e);
                }
            }
        }
        return true;
    }
}

Note: this code assumes s3Client and log are declared and initialized elsewhere. Method returns a boolean, but can throw exceptions.

Answer 11

As others have mentioned, for the AWS S3 Java SDK 2.10+ you can use the HeadObjectRequest object to check if there is a file in your S3 bucket. This will act like a GET request without actually getting the file.

Example code since others haven't actually added any code above:

public boolean existsOnS3 () throws Exception {
    try {
       S3Client s3Client = S3Client.builder ().credentialsProvider (...).build ();
       HeadObjectRequest headObjectRequest = HeadObjectRequest.builder ().bucket ("my-bucket").key ("key/to/file/house.pdf").build ();
       HeadObjectResponse headObjectResponse = s3Client.headObject (headObjectRequest);
       return headObjectResponse.sdkHttpResponse ().isSuccessful ();    
   }
   catch (NoSuchKeyException e) {
      //Log exception for debugging
      return false;
   }
}

Answer 12

Using Object isting. Java function to check if specified key exist in AWS S3.

boolean isExist(String key)
    {
        ObjectListing objects = amazonS3.listObjects(new ListObjectsRequest().withBucketName(bucketName).withPrefix(key));

        for (S3ObjectSummary objectSummary : objects.getObjectSummaries())
        {
            if (objectSummary.getKey().equals(key))
            {
                return true;
            }

        }
        return false;
    }

Answer 13

There is an easy way to do it using jetS3t API's isObjectInBucket() method.

Sample code:

ProviderCredentials awsCredentials = new AWSCredentials(
                awsaccessKey,
                awsSecretAcessKey);

        // REST implementation of S3Service
        RestS3Service restService = new RestS3Service(awsCredentials);

        // check whether file exists in bucket
        if (restService.isObjectInBucket(bucket, objectKey)) {

            //your logic

        }

Answer 14

I also faced this problem when I used

String BaseFolder = "3patti_Logs"; 
S3Object object = s3client.getObject(bucketName, BaseFolder);
 

I got error key not found

When I hit and try

String BaseFolder = "3patti_Logs"; 
S3Object object = s3client.getObject(bucketName, BaseFolder+"/");

it worked , this code is working with 1.9 jar otherwise update to 1.11 and use doesObjectExist as said above

Answer 15

Alternatively you can use Minio-Java client library, its Open Source and compatible with AWS S3 API.

You can use Minio-Java StatObject.java examples for the same.

import io.minio.MinioClient;
import io.minio.errors.MinioException;

import java.io.InputStream;
import java.io.IOException;
import java.security.NoSuchAlgorithmException;
import java.security.InvalidKeyException;

import org.xmlpull.v1.XmlPullParserException;


public class GetObject {
  public static void main(String[] args)
    throws NoSuchAlgorithmException, IOException, InvalidKeyException, XmlPullParserException, MinioException {
    // Note: YOUR-ACCESSKEYID, YOUR-SECRETACCESSKEY and my-bucketname are
    // dummy values, please replace them with original values.
    // Set s3 endpoint, region is calculated automatically
    MinioClient s3Client = new MinioClient("https://s3.amazonaws.com", "YOUR-ACCESSKEYID", "YOUR-SECRETACCESSKEY");
    InputStream stream = s3Client.getObject("my-bucketname", "my-objectname");

    byte[] buf = new byte[16384];
    int bytesRead;
    while ((bytesRead = stream.read(buf, 0, buf.length)) >= 0) {
      System.out.println(new String(buf, 0, bytesRead));
    }

    stream.close();
  }
}

I hope it helps.

Disclaimer : I work for Minio

Answer 16

Use the jets3t library. Its a lot more easier and robust than the AWS sdk. Using this library you can call, s3service.getObjectDetails(). This will check and retrieve only the details of the object (not the contents) of the object. It will throw a 404 if the object is missing. So you can catch that exception and deal with it in your app.

But in order for this to work, you will need to have ListBucket access for the user on that bucket. Just GetObject access will not work. The reason being, Amazon will prevent you from checking for the presence of the key if you dont have ListBucket access. Just knowing whether a key is present or not, will also suffice for malicious users in some cases. Hence unless they have ListBucket access they will not be able to do so.