AWS java maven project to retrieve all the S3 keys

Question

I have created a maven project for my AWS java development. I'm trying to get the list of all the files in a S3 bucket with a given prefix.

Here is my code:

app.java:

import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.*;
import com.amazonaws.services.s3.transfer.Copy;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.TransferManagerBuilder;

import java.io.InputStream;
import java.util.LinkedList;
import java.util.List;

class Main {

    public static List<String> getObjectsListFromS3(AmazonS3 s3, String bucket, String prefix) {
        final String delimiter = "/";
        if (!prefix.endsWith(delimiter)) {
            prefix = prefix + delimiter;
        }
    
        List<String> paths = new LinkedList<>();
        ListObjectsRequest request = new ListObjectsRequest().withBucketName(bucket).withPrefix(prefix);
    
        ObjectListing result;
        do {
            result = s3.listObjects(request);
    
            for (S3ObjectSummary summary : result.getObjectSummaries()) {
                // Make sure we are not adding a 'folder'
                if (!summary.getKey().endsWith(delimiter)) {
                    paths.add(summary.getKey());
                }
            }
            request.setMarker(result.getMarker());
        }
        while (result.isTruncated());
    
        return paths;
    }

    public static void main(String[] args) {
        String bucket = "playground-us-east-1-1234567890";
        AmazonS3 s3 = AmazonS3ClientBuilder.standard().withRegion("us-east-1").build();
    
        String prefix = "test";
        for (String key : getObjectsListFromS3(s3, bucket, prefix)) {
            System.out.println(key);
        }
    }
}

Here is my pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example.myapp</groupId>
  <artifactId>myapp</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>myapp</name>
  <url>http://maven.apache.org</url>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

When I try to run the code mvn package, I'm getting the following error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project myapp: Compilation failure
[ERROR] /home/user/Desktop/myapp/src/main/java/com/example/myapp/App.java:[35,41] diamond operator is not supported in -source 1.5
[ERROR]   (use -source 7 or higher to enable diamond operator)
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Before, I was getting something similar error but I changed my pom.xml to java version1 compatible. But I don't understand this error and where I'm going wrong

Does this answer your question? [Specifying java version in maven - differences between properties and compiler plugin](https://stackoverflow.com/questions/38882080/specifying-java-version-in-maven-differences-between-properties-and-compiler-p) — luk2302, Aug 09 '21 at 20:44

smac2020 · Accepted Answer · 2021-08-09T21:12:48.107

You are using an older API for Amazon S3. You should considering moving to Java API V2. V1 package names are:

   **com.amazonaws.services..**.

while V2 package names are:

  **software.amazon.awssdk.services...**

Amazon strongly recommends moving to V2:

The AWS SDK for Java 2.x is a major rewrite of the version 1.x code base. It’s built on top of Java 8+ and adds several frequently requested features. These include support for non-blocking I/O and the ability to plug in a different HTTP implementation at run time.

You can find the POM file with the required dependencies in this Github repo:

https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/javav2/example_code/s3

All S3 examples are located here too.

UPDATE

Perform these Steps to get an Amazon S3 Java V2 code example (ListObjects) working:

Create a new Java Maven project in an IDE like IntelliJ.

Copy the POM file from the link above and drop it into your POM file.

Create a package named demo.

In the demo package, create a class named ListObjects.

Copy the code from here:

https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/ListObjects.java

Add this code to your ListObjects class.

Run the code and see the output in the IDE:

I'm importing ```import com.amazonaws.services.s3.model.*;``` but still I'm getting Compilation failure: ```package com.amazonaws.regions does not exist```, ```ackage com.amazonaws.services.s3 does not exist``` and so on.. — R0bert, Aug 09 '21 at 21:02
ALso - feel free to connect to me in Linked IN. My Linked IN is under my profile. Then once connected, you can ask me Java API questions anytime in that tool. — smac2020, Aug 09 '21 at 21:22
I sent an invite to connect. I tried the above steps but still facing the same issue. ```package software.amazon.awssdk.regions does not exist``` — R0bert, Aug 09 '21 at 22:43

AWS java maven project to retrieve all the S3 keys

1 Answers1