Delete files, directories and buckets in amazon s3 java

Question

I'm wondering how to do this. I looked at the sdk documentation and have some examples, but am confused how the syntax generally goes.

If I want to delete a file, I assume I use deleteObject(path, key). However, what is the "key"?

Also how do you delete a directory? I can't seem to find a method for doing that.

Only an empty directory can be deleted. So, first delete the contents and then delete the directory using s3Client.deleteObject(bucket name, folderName). — Ankush, Jun 25 '19 at 08:52
[Refer this](https://forums.aws.amazon.com/message.jspa?messageID=225866) — Ankush, Jun 25 '19 at 09:03

score 47 · Answer 1 · edited Jan 16 '14 at 08:30

47

This snippet of code works for me. folderPath is something like "topDir/secondDir/"

void deleteObjectsInFolder(String bucketName, String folderPath) {
   for (S3ObjectSummary file : s3.listObjects(bucketName, folderPath).getObjectSummaries()){
      s3.deleteObject(bucketName, file.getKey());
    }
}

edited Jan 16 '14 at 08:30

Martin Prikryl

188,800
56
490
992

answered Nov 16 '13 at 05:43

Danger

2,043
2
21
24

Could you provide more information on why this code doesn't work for you? I copied it from my running code. – Danger Jul 23 '14 at 07:34
11

careful that getObjectSummaries can return a truncated list ! – CanardMoussant Nov 28 '16 at 16:09
Anyone have an idea how to do this with V2? – markthegrea Jan 21 '20 at 21:33

Brendan Long · Accepted Answer · 2019-04-30T15:48:12.173

16

A "key" in S3 is similar to a file path:

http://bucket.s3.amazonaws.com/some/path/to/use

... is in a bucket named bucket and has a key of some/path/to/use.

It's not actually a path though, because there are no folders. The S3 key is just the file name for a file in one big directory (the entire bucket). S3 keys can contain /, but it has no special meaning unless you set the delimiter argument with listing a bucket.

In other words, having an object named some/object doesn't tell you anything about the object some (it might or might not exist -- the two objects are not related).

However, you can request keys with a specific prefix, so I could say "give me all keys starting with some/path/to/ and it will return some/path/to/use. It looks like "listing a directory", but it's really just asking for files that start with a specific string of characters.

I could just as easily name things like this:

somepathtousea
somepathtouseb

And say "give me everything starting with somepathtouse" (and it would say somepathtousea and somepathtouseb).

Note: S3 URL's come in several forms:

http://s3.amazonaws.com/bucket/key
http://bucket.s3.amazonaws.com/key
http://bucket/key (where bucket is a DNS CNAME record pointing to bucket.s3.amazonaws.com)

EDIT:

I looked at the JavaDocs and this is the function signature I see (for AmazonS3Client):

public void deleteObject(java.lang.String bucketName,
                         java.lang.String key)
                  throws AmazonClientException,
                         AmazonServiceException

EDIT again:

Folders do kind-of exist now, as zero-length objects with a content-type of application/x-directory and a key ending in /:

$ AWS_PROFILE=prod aws s3api head-object --bucket example-bucket --key example-directory/
{
    "AcceptRanges": "bytes",
    "LastModified": "Mon, 29 Apr 2019 14:59:36 GMT",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "ContentType": "application/x-directory",
    "ServerSideEncryption": "AES256",
    "Metadata": {}
}

This is still just convention and there's nothing stopping you from having files ending / or files inside of "folders" that don't exist.

edited Apr 30 '19 at 15:48

answered Oct 14 '11 at 05:07

Brendan Long

53,280
21
146
188

so really the syntax should be `deleteObject(bucketName, path)`? – locoboy Oct 14 '11 at 05:10
@cfarm54 - Except a key isn't a path. They key-bucket relationship is more like.. the key is a file name (which may contain `/`) and the bucket is one big folder. – Brendan Long Oct 14 '11 at 05:13
Thanks, and how would you delete the directory and bucket? – locoboy Oct 14 '11 at 05:47
@cfarm54 - What I'm trying to get across is that there are no directories. You can't delete one because they don't exist. To delete a bucket, I suspect you would call `AmazonS3Client.deleteBucket(String bucketname)` – Brendan Long Oct 14 '11 at 05:49
but lets say that your file structure was /mybucket/dir1/dir2 and in dir2 you have a set of files. Right now you covered how to delete individual files, but what if i want to delete dir2 or dir1? and then the next question is how do i delete the whole `mybucket`? – locoboy Oct 14 '11 at 05:59
2

@cfarm54 - Look at the [documentation for AmazonS3Client](http://docs.amazonwebservices.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html). `deleteBucket(String bucketName)` deletes an entire bucket. [S3 has no method to delete keys with a common prefix](http://stackoverflow.com/questions/3178685/how-to-delete-lot-of-objects-named-with-common-prefix-from-s3-bucket). You'll have to list them (`listObjects(String bucketName, String prefix)`) and then delete them individually. – Brendan Long Oct 14 '11 at 06:04
@BrendanLong if there are only object keys and no directories, how is it that the CLI has recursive option, and that the .Net SDK allows to delete directories? https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/S3/MS3DirectoryInfoDeleteBoolean.html – Крис Apr 15 '19 at 06:09
S3 lets you request files by prefix, which is kind of like listing a directory, but these are two differences: directories don't exist, so you don't need to create them, and `/` isn't special: you can do something like `aws s3 ls my-bucket/some-prefix` and it will find files like `my-bucket/some-prefix.json` – Brendan Long Apr 15 '19 at 16:45
If you do `aws s3 ls my-bucket/some-dir/` you will find files like `my-bucket/some-dir/file1.txt` but you will not see `my-bucket/some-dir/some-sub-dir/file2.txt`, unless you use the `--recursive` option. So `/` is special for the CLI. Not for S3 as an object store, but for the AWS CLI. And did you look at the .Net SDK? It has a `S3DirectoryInfo` class. The Java SDK seems more limited. It looks like a missing a feature. – Крис Apr 18 '19 at 17:40
1

Ah I see, if you set `delimiter` to `/`, ListBucket will sort-of treat `/`'s in keys as directory separators: https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html I assume awscli sets this by default. – Brendan Long Apr 18 '19 at 18:19

score 8 · Answer 3 · answered Oct 14 '11 at 05:15

You might want to take a look at this example for a quick reference on how you can delete objects from S3.

The syntax for delete is actually deleteObject( bucketName, key ) where bucketName is the bucket in which you have placed your files and key is name of the file you want to delete within the bucket.

Think of a bucket as your hard disk drive like C:\ , D:\ etc. And key as the absolute pathname of a file you want to delete.

Tariq Kamal · Answer 4 · 2015-11-10T19:43:12.633

6

/*Here is solution that works for me. Here Bucket_Name is my bucket name on S3, and key is the path under Bucket_Name. So, if absolute path on S3 is:

s3://my_bucket/Path/to/my/folder

then, the code below should work. */

    String Bucket_Name = "my_bucket";
    String key = "Path/to/my/folder";   
    ObjectListing objects = s3Client.listObjects(BUCKET_NAME, key);
        for (S3ObjectSummary objectSummary : objects.getObjectSummaries()) 
            {
            s3Client.deleteObject(BUCKET_NAME, objectSummary.getKey());
            }

edited Nov 10 '15 at 19:43

answered Oct 28 '15 at 19:42

Tariq Kamal

468
1
8
17

4

Welcome to SO. You haven't addressed the question of "what is the key". You are supplying code but it needs some explanation to make it higher value for other users with questions. – Shawn Mehan Oct 28 '15 at 22:41
@ShawnMehan: Thanks for your feedback, I will update my answer. – Tariq Kamal Nov 10 '15 at 19:44

score 4 · Answer 5 · answered Nov 13 '17 at 17:03

As question is asking about Deleting files, directories and buckets in amazon S3 java, I would like to offer code for deleting a non-empty S3 bucket (AWS Reference):

 public void deleteBucket(final String bucketName) {

        final AmazonS3 s3 = AmazonS3ClientBuilder.defaultClient();

        try {
            ObjectListing objectListing = s3.listObjects(bucketName);
            while (true) {
                for (Iterator<?> iterator = objectListing.getObjectSummaries().iterator(); iterator.hasNext(); ) {
                    S3ObjectSummary summary = (S3ObjectSummary) iterator.next();
                    s3.deleteObject(bucketName, summary.getKey());
                }

                if (objectListing.isTruncated()) {
                    objectListing = s3.listNextBatchOfObjects(objectListing);
                } else {
                    break;
                }
            }

            VersionListing versionListing = s3.listVersions(new ListVersionsRequest().withBucketName(bucketName));
            while (true) {
                for (Iterator<?> iterator = versionListing.getVersionSummaries().iterator(); iterator.hasNext(); ) {
                    S3VersionSummary vs = (S3VersionSummary) iterator.next();
                    s3.deleteVersion(bucketName, vs.getKey(), vs.getVersionId());
                }

                if (versionListing.isTruncated()) {
                    versionListing = s3.listNextBatchOfVersions(versionListing);
                } else {
                    break;
                }
            }

            s3.deleteBucket(bucketName);
        } catch (AmazonServiceException e) {
            System.err.println(e.getErrorMessage());
        }
    }

score 2 · Answer 6 · answered Feb 28 '14 at 01:35

2

Works for me, beware of truncation!

    long start = System.currentTimeMillis();
    long totalSize = 0;
    int totalItems = 0;

    String key ="path/to/folder/"
    String bucket = "my-bucket"

    final ListObjectsRequest listObjectsRequest = new ListObjectsRequest().withBucketName(bucketName).withPrefix(key);

    ObjectListing objects = s3.listObjects(listObjectsRequest);
    do {
        for (S3ObjectSummary objectSummary : objects.getObjectSummaries()) {
            totalSize += objectSummary.getSize();
            totalItems++;
            s3.deleteObject(bucketName, objectSummary.getKey());
        }
        objects = s3.listNextBatchOfObjects(objects);
    } while (objects.isTruncated());

    long stop = System.currentTimeMillis();

    LOG.trace("User {} had {} items with {} Kb, took {} ms to delete", user.getName(), totalItems, totalSize / 1024, stop
            - start);

answered Feb 28 '14 at 01:35

Papick G. Taboada

565
5
15

Have you tested this code? Cause I would say that in case of 'truncation' this code will not delete the last batch of files read by the `listNextBatchOfObjects`. Another thing is that I'm wondering if Amazon will properly handle listing objects in batches while they are being deleted in the same time? – MJar Sep 27 '16 at 12:12
As mentioned by @MJar, this code is dangerous and will not work as intended. – david Nov 07 '17 at 22:43
You need to assign the new value after the isTruncated. Something like this : ```} while (objects.isTruncated() && (objects = s3.listNextBatchOfObjects(objects)) != null);``` – Fabich Dec 19 '18 at 12:19

score 2 · Answer 7 · edited May 12 '18 at 09:38

The ListObjectsV2Result worked for me. Try once.

private void deleteObjectsInFolder() {
    try {
        ListObjectsV2Result result;
        do {
            String folderPath =   " ";



            result = s3.listObjectsV2(Constants.BUCKET_NAME, folderPath);

            Log.e("count:", result.getKeyCount() + "");

            if (result.getKeyCount() != 0) {

                for (S3ObjectSummary objectSummary :
                        result.getObjectSummaries()) {
                    s3.deleteObject(Constants.BUCKET_NAME, objectSummary.getKey());
                }

            }


            System.out.println("Next Continuation Token : " + result.getNextContinuationToken());
        } while (result.isTruncated() == true);

    } catch (AmazonServiceException ase) {
        System.out.println("Caught an AmazonServiceException, " +
                "which means your request made it " +
                "to Amazon S3, but was rejected with an error response " +
                "for some reason.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("AWS Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    } catch (AmazonClientException ace) {
        System.out.println("Caught an AmazonClientException, " +
                "which means the client encountered " +
                "an internal error while trying to communicate" +
                " with S3, " +
                "such as not being able to access the network.");
        System.out.println("Error Message: " + ace.getMessage());
    }
}

Not efficient as will send calls to S3 for each of the folder delete, better to use `DeleteObjectsRequest` and then delete — Prashant Shubham, Feb 11 '22 at 08:36

score 2 · Answer 8 · answered Jun 24 '18 at 06:20

Deleting a list of objects from S3 bucket by bulks:

public void deleteObjects(String bucketName, List<String> keys) {

    List<KeyVersion> bulk = new ArrayList<>();
    for (int i = 0; i < keys.size(); i++) {
        bulk.add(new KeyVersion(keys.get(i)));
        if (i % 100 == 0) {
            try {
                s3Client.deleteObjects(new DeleteObjectsRequest(bucketName).withKeys(bulk));
            } catch (Exception e) {
                System.err.println(e.getErrorMessage());
            }
            bulk.clear();
        }
    }
    if (bulk.size() > 0) {
        try {
            s3Client.deleteObjects(new DeleteObjectsRequest(bucketName).withKeys(bulk));
        } catch (Exception e) {
            System.err.println(e.getErrorMessage());
        }
    }
}

Source: http://codeflex.co/delete-objects-from-amazon-s3-bucket-using-aws-sdk-for-java/

score 2 · Answer 9 · edited Aug 16 '18 at 19:16

2

This line of code works in my case where the keyName is the file name:

s3Client.deleteObject(new DeleteObjectRequest(bucketName, keyName));

edited Aug 16 '18 at 19:16

Azat Ibrakov

9,998
9
38
50

answered Aug 16 '18 at 16:17

Philippe

495
1
7
18

score 0 · Answer 10 · answered Jan 23 '23 at 21:23

kotlin

class S3(
    var bucketName: String? = null,
    var key: String? = null,
    val accessKey: String? = null,
    val secretKey: String? = null,
    val region: String? = null
)

fun delete(
    s3: S3,
    keyword: String = "",
) {
    with(s3) {
        val client = client(accessKey, secretKey, region)

        var objects = client.listObjects(bucketName, key)
        while (true) {
            for (i in objects.objectSummaries) {
                if (!i.key.contains(keyword)) {
                    continue
                }
                client.deleteObject(bucketName, i.key)
            }

            if (objects.isTruncated) {
                objects = client.listNextBatchOfObjects(objects)
            } else {
                break
            }
        }

        var versions = client.listVersions(bucketName, key)
        while (true) {
            for (i in versions.versionSummaries) {
                if (!i.key.contains(keyword)) {
                    continue
                }
                client.deleteVersion(bucketName, i.key, i.versionId)
            }

            if (versions.isTruncated) {
                versions = client.listNextBatchOfVersions(versions)
            } else {
                break
            }
        }
    }
}

score 0 · Answer 11 · answered Mar 20 '23 at 22:34

Here's my answer which worked for me, I used the answer from Danger

This takes time but eventually it deletes all the files and folder.

public static void deleteObjects(AmazonS3 s3Client, String bucketName, String folderPath)
    {
    while(getFilesListInBucket(s3Client, bucketName, folderPath).size()>0) {
        for (S3ObjectSummary file : s3Client.listObjects(bucketName, folderPath).getObjectSummaries()) {
            s3Client.deleteObject(bucketName, file.getKey());
        }
    }
}

Delete files, directories and buckets in amazon s3 java

11 Answers11

kotlin