0

Is there any way to do a bulk delete in the v1 AWS SDK that does the same as this CLI call?

$ aws s3 rm s3://my-bucket --dryrun --recursive --exclude "*" --include "PREFIX_0/*"
(dryrun) delete: s3://my-bucket/PREFIX_0/file_0.txt
(dryrun) delete: s3://my-bucket/PREFIX_0/file_1.txt
(dryrun) delete: s3://my-bucket/PREFIX_0/file_2.txt
(dryrun) delete: s3://my-bucket/PREFIX_0/file_3.txt

The SDK only offers deleteObjects which requires a list of keys (not prefixes), so it must be used in conjunction with listObjects

private static void deleteObjects(AmazonS3 client, String bucketName, List<String> objects) {
    DeleteObjectsRequest request = new DeleteObjectsRequest(bucketName);
    request.withKeys(objects.toArray(new String[]{}));
     client.deleteObjects(request);
}

listObjects is limited to 1000 keys per result, so you'd have to iterate and use markers as a kind of "pagination". Also not sure how well deleteObjects with a high number of keys performs.

private static List<String> listObjects(AmazonS3 client, String bucketName, Optional<String> prefix) {
    boolean checkForObjects = true;

    List<String> objectResultList = new ArrayList<>();
    ListObjectsRequest listRequest = new ListObjectsRequest();

    while (checkForObjects){
        // check for objects with a given prefix
        listRequest.withBucketName(bucketName);
        prefix.ifPresent(listRequest::withPrefix);
        ObjectListing objects = client.listObjects(listRequest);
        // only do another run if result was truncated
        checkForObjects = objects.isTruncated();

        // use next marker to get next set of results
        listRequest.setMarker(objects.getNextMarker());

        List<String> objectList = objects.getObjectSummaries()
                .stream()
                .map(S3ObjectSummary::getKey)
                .collect(Collectors.toList());

        objectResultList.addAll(objectList);
    }

    return objectResultList;
}
DoNuT
  • 463
  • 1
  • 7
  • 23
  • 2
    Not that I'm aware of. You'd have to implement the same high-level feature that the awscli implements. Also see: https://stackoverflow.com/questions/42442259/delete-a-folder-and-its-content-aws-s3-java – jarmod Aug 11 '22 at 15:00
  • 1
    @jarmod: Thanks, I just ran the CLI operation with `--debug` option and it really seems that it's actually running listObjects, then applies filters locally and delete calls, so no more efficient than code sample above. – DoNuT Aug 16 '22 at 10:18

0 Answers0