How can I read an AWS S3 File with Java?

Question

I tried to read a file from AWS S3 to my java code:

  File file = new File("s3n://mybucket/myfile.txt");
  FileInputStream fileInput = new FileInputStream(file);

Then I got an error:

java.io.FileNotFoundException: s3n:/mybucket/myfile.txt (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)

Is there a way to open/read a file from AWS S3? Thanks a lot!

Answer 1

The 'File' class from Java doesn't understand that S3 exists. Here's an example of reading a file from the AWS documentation:

AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());        
S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, key));
InputStream objectData = object.getObjectContent();
// Process the objectData stream.
objectData.close();

Answer 2

In 2019 there's a bit more optimal robust way to read a file from S3:

private final AmazonS3 amazonS3Client = AmazonS3ClientBuilder.standard().build();

private Collection<String> loadFileFromS3() {
    try (final S3Object s3Object = amazonS3Client.getObject(BUCKET_NAME,
                                                            FILE_NAME);
        final InputStreamReader streamReader = new InputStreamReader(s3Object.getObjectContent(), StandardCharsets.UTF_8);
        final BufferedReader reader = new BufferedReader(streamReader)) {
        return reader.lines().collect(Collectors.toSet());
    } catch (final IOException e) {
        log.error(e.getMessage(), e)
        return Collections.emptySet();
    }
}

Do note the comment about the memory usage of this piece of code.

Answer 3

We can also use software.amazon.awssdk:s3

 //Assuming the credentials are read from Environment Variables, so no hardcoding here

    S3Client client = S3Client.builder()
                        .region(regionSelected)
                        .build();
    
    GetObjectRequest getObjectRequest = GetObjectRequest.builder()
                    .bucket(bucketName)
                    .key(fileName)
                    .build();
    
    ResponseInputStream<GetObjectResponse> responseInputStream = client.getObject(getObjectRequest);

    InputStream stream = new ByteArrayInputStream(responseInputStream.readAllBytes());
    
    
    System.out.println("Content :"+ new String(responseInputStream.readAllBytes(), StandardCharsets.UTF_8));

Answer 4

Steps to read S3 file in java can be:

1. Create AmazonS3Client.
2. Create S3Object using bucket name and key.
3. Create buffer reader using S3Object and read file line by line.

1 >>>

    BasicAWSCredentials awsCreds = new BasicAWSCredentials("accessKey", "secretKey");
    AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
            .withCredentials(new AWSStaticCredentialsProvider(awsCreds))
            .withRegion("region_name_here").build();  

2 >>>

   S3Object object = s3Client.getObject(new GetObjectRequest("bucketName", "key"));

3 >>>

   BufferedReader reader = new BufferedReader(new InputStreamReader(object.getObjectContent()));

    String s = null;
    while ((s = reader.readLine()) != null)
    {
        System.out.println(s);
        //your business logic here
    }

Thanks.

Answer 5

If the file's content is a string, then you can use getObjectAsString. Otherwise, you can use IOUtils.toByteArray on getObjectContent() to read the file's content into a byte array.

Obviously, these are best used on small-ish S3 objects that will easily fit into memory.

private final AmazonS3 amazonS3Client = AmazonS3ClientBuilder.standard().build();

private String loadStringFromS3() {
    try {
        return amazonS3Client.getObjectAsString(BUCKET_NAME, FILE_NAME);
    } catch (final IOException e) {
        log.error(e.getMessage(), e)
        return null;
    }
}

private byte[] loadDataFromS3() {
    try (final S3Object s3Object = amazonS3Client.getObject(BUCKET_NAME, FILE_NAME)) {
        return IOUtils.toByteArray(s3Object.getObjectContent());
    } catch (final IOException e) {
        log.error(e.getMessage(), e)
        return null;
    } finally {
        IOUtils.closeQuietly(object, log);
    }
}

Answer 6

This is my solution. I'm using spring boot 2.4.3

Create an amazon s3 client

AmazonS3 amazonS3Client = AmazonS3ClientBuilder
                .standard()
                .withRegion("your-region")
                .withCredentials(
                        new AWSStaticCredentialsProvider(
                            new BasicAWSCredentials("your-access-key", "your-secret-access-key")))
                .build();

Create an amazon transfer client.

TransferManager transferManagerClient = TransferManagerBuilder.standard()
                .withS3Client(amazonS3Client)
                .build();

Create a temporary file in /tmp/{your-s3-key} so that we can put the file we download in this file.

File file = new File(System.getProperty("java.io.tmpdir"), "your-s3-key"); 

try {
    file.createNewFile(); // Create temporary file
} catch (IOException e) {
    e.printStackTrace();
}

file.mkdirs();  // Create the directory of the temporary file

Then, we download the file from s3 using transfer manager client

// Note that in this line the s3 file downloaded has been transferred in to the temporary file that we created
Download download = transferManagerClient.download(
               new GetObjectRequest("your-s3-bucket-name", "your-s3-key"), file); 

// This line blocks the thread until the download is finished
download.waitForCompletion();  

Now that the s3 file has been successfully transferred into the temporary file that we created. We can get the InputStream of the temporary file.

InputStream input = new DataInputStream(new FileInputStream(file));

Because the temporary file is not needed anymore, we just delete it.

file.delete();

Answer 7

This is the correct way to do it as of 2023:

  public Long downloadFile(S3TransferManager transferManager, String bucketName,
                             String key, String downloadedFileWithPath) {
        DownloadFileRequest downloadFileRequest =
                DownloadFileRequest.builder()
                        .getObjectRequest(b -> b.bucket(bucketName).key(key))
                        .addTransferListener(LoggingTransferListener.create())
                        .destination(Paths.get(downloadedFileWithPath))
                        .build();

        FileDownload downloadFile = transferManager.downloadFile(downloadFileRequest);

        CompletedFileDownload downloadResult = downloadFile.completionFuture().join();
        log.info("Content length [{}]", downloadResult.response().contentLength());
        return downloadResult.response().contentLength();
    }

Answer 8

If you are using JDK 1.8 or later and Java NIO then you can make use of the aws-java-nio-spi-for-s3 package which provides a drop-in SPI that will allow Java to read and write S3 objects using standard Java Paths, Channels etc.

https://github.com/awslabs/aws-java-nio-spi-for-s3