How can I unit test the getObject method of the AWS S3 SDK using Java?

Question

I'm working with Java and I'm using the AWS SDK for interact with S3. I've the following method and I want to unit test it

private final S3Client s3Client;
...
...
public byte[] download(String key) throws IOException {
    GetObjectRequest getObjectRequest = GetObjectRequest.builder()
            .bucket("myBucket")
            .key(key)
            .build();
    return s3Client.getObject(getObjectRequest).readAllBytes();
}

For this purpose I'm using JUnit 5 and Mockito. The problem is that I don't know how to mock the result of

s3Client.getObject(getObjectRequest)

because the return type

ResponseInputStream<GetObjectResponse>

is a final class.

Any idea or suggestions? Thank you

If you can not simply create a matching `ResponseInputStream` object, you probably want to enable final mocking as described in the [docu](https://javadoc.io/doc/org.mockito/mockito-core/latest/org/mockito/Mockito.html). Are you sure about the class though, I can't see a `readAllBytes()` method on that class (at least in the [`latest version`](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/core/ResponseInputStream.html)) ? — second, Nov 28 '19 at 20:45
s3Client.getObject(getObjectRequest) returns a ResponseInputStream which undirectly extends java.io.InputStream — Gavi, Nov 29 '19 at 23:24
Apparently the method was added in java 9 while the documentation I linked refers to an older java version which did not have that method. — second, Nov 30 '19 at 09:14

score 8 · Answer 1 · answered Nov 24 '20 at 11:54

In case anyone is still looking for a different solution this is how I did it:
This is the code that needs to be mocked:

InputStream objectStream =
    this.s3Client.getObject(
        GetObjectRequest.builder().bucket(bucket).key(key).build(),
        ResponseTransformer.toInputStream());

This is how to mock it:

S3Client s3Client = Mockito.mock(S3Client.class);
String bucket = "bucket";
String key = "key";
InputStream objectStream = getFakeInputStream();
when(s3Client.getObject(
        Mockito.any(GetObjectRequest.class),
        ArgumentMatchers
            .<ResponseTransformer<GetObjectResponse, ResponseInputStream<GetObjectResponse>>>
                any()))
    .then(
        invocation -> {
          GetObjectRequest getObjectRequest = invocation.getArgument(0);
          assertEquals(bucket, getObjectRequest.bucket());
          assertEquals(key, getObjectRequest.key());

          return new ResponseInputStream<>(
              GetObjectResponse.builder().build(), AbortableInputStream.create(objectStream));
        });

score 3 · Accepted Answer · answered Apr 18 '20 at 21:51

The problem is solved. In a maven project you can add a file named "org.mockito.plugins.MockMaker" in the folder "src/test/resources/mockito-extensions".

Inside the file, add "mock-maker-inline" without quotes.

From now Mockito will be able to mock final classes also.

score 2 · Answer 3 · answered Jan 21 '20 at 05:13

I was able to get this going in Spock with a GroovyMock that uses Objenesis. I know it's not the original poster's stack but this came up in my search so I thought I'd respond here in case parts of this help anyone else.

S3Repo to Test

import software.amazon.awssdk.services.s3.S3Client
import software.amazon.awssdk.services.s3.model.GetObjectRequest

class S3Repo {
    S3Client s3
    String bucket

    def getS3ObjectText(String key) {
        def response
        try {
            response = s3.getObject(GetObjectRequest
                    .builder().bucket(bucket).key(key).build() as GetObjectRequest)
            return response.text
        } finally {
            if (response) response.close()
        }
    }
}

Spock Test

import software.amazon.awssdk.core.ResponseInputStream
import software.amazon.awssdk.services.s3.S3Client
import software.amazon.awssdk.services.s3.model.GetObjectRequest
import spock.lang.Specification

class S3RepoTest extends Specification {

    def "get object text"() {
        given:
        def response = GroovyMock(ResponseInputStream)
        def s3Text = 'this would be in file stored in s3'
        def key = 'my-file.txt'
        def s3 = Mock(S3Client)
        def bucket = 'mock-bucket'
        def repo = new S3Repo(s3: s3, bucket: bucket)

        when:
        def text = repo.getS3ObjectText(key)

        then:
        1 * s3.getObject(_) >> { args ->
            def req = args.first() as GetObjectRequest
            assert req.bucket() == bucket
            assert req.key() == key
            return response
        }
        1 * response.text >> s3Text

        and:
        text == s3Text
    }
}

I think the critical piece here is the GroovyMock which requires Objenesis. You can certainly test your Java code with Groovy and you could probably use the GroovyMock in JUnit.

Zhang Mingxiang · Answer 4 · 2022-07-21T10:30:04.167

You're going to mock S3Client instead of ResponseInputStream.

  @Mock
  S3Client s3;


  String storageKey = "123";
  byte[] data = new byte[] {11, 22, 33, 44, 55, 66, 77, 88, 99};


  @Test
  public void testGet() throws Exception {
    ResponseInputStream<GetObjectResponse> res =new ResponseInputStream(
            GetObjectResponse.builder().build(),
            AbortableInputStream.create(new ByteArrayInputStream(data)));
    when(s3.getObject(GetObjectRequest.builder().bucket(bucketName).key(storageKey).build()))
            .thenReturn(res);

    byte[] dataRetrieved = targetInstance.download(storageKey); // virtual code
    assertThat(dataRetrieved).containsExactly(data);
  }

score 1 · Answer 5 · answered Jul 05 '20 at 09:15

I did this in a gradle project. Add the following dependencies -

    testCompile group: 'net.bytebuddy', name: 'byte-buddy-agent', version: '1.9.10'
    testCompile group: 'org.mockito', name: 'mockito-inline', version: '2.7.21'
    compile group: 'net.bytebuddy', name: 'byte-buddy', version: '1.9.10'

The bytebuddy dependencies are compile dependencies required for mockito-core to support mockito-inline dependencies. (Ref - https://howtodoinjava.com/mockito/plugin-mockmaker-error/ https://mvnrepository.com/artifact/org.mockito/mockito-core/2.27.2 (see compile dependencies here))

Was able to mock the getObject method after this.

score 0 · Answer 6 · answered Jan 17 '22 at 12:57

I have an addition to all the great answers here.

There is an opinion that developers should not mock the types and APIs they don't own. Here are a few links on this topic:

TLDR: by mocking you introduce some assumptions of how things should work into your tests. So your tests are not really black-box anymore, while they should be. Your assumptions may be wrong. Or the API you've mocked could change, making your mocks and tests drift and stale. This could result in a broken code passing the tests.

Need an example? Consider this code:

@Test
void test() throws Exception {
    GetObjectRequest request = GetObjectRequest
        .builder()
        .bucket("bucket")
        .key("file.json")
        .range("bytes=0-0,-1")
        .build();
    ResponseInputStream<GetObjectResponse> response = client.getObject(request);
    byte[] bytes = response.readAllBytes();

    Assertions.assertEquals("{}", new String(bytes));
}

One would expect it to successfully read first and last bytes of a JSON file. Which would give {} (assuming there are no extra whitespaces or newlines). So she mocks the getObject and returns those bytes.

What happens in the reality? She'll get the whole JSON, not just the first and last bytes, because

Amazon S3 doesn't support retrieving multiple ranges of data per GET request.

This is just how it works in the real world: no exceptions, no warnings, just the whole content being returned.

If you leave only one range, let's say only the first byte, it would work:

@Test
void test() throws Exception {
    GetObjectRequest request = GetObjectRequest
        .builder()
        .bucket("bucket")
        .key("file.json")
        .range("bytes=0-0")
        .build();
    ResponseInputStream<GetObjectResponse> response = client.getObject(request);
    byte[] bytes = response.readAllBytes();

    Assertions.assertEquals("{", new String(bytes));
}

Although this example is completely made-up, and cases like this are rare, I hope you've got the idea.

Solution?

Integration tests! Yes, it's a higher level and writing them is more effort, but they make it possible to catch bugs like the one above.

So, next time think about writing an integration test and using a real thing instead of a mock. For some technologies you would even find an official lightweight implementations. Like DynamoDB Local. Or use a mock that is closer to the original API. Like a LocalStack.

And if you are a lucky user of Java/Kotlin and JUnit 5 I would like to recommend you to use aws-junit5, a set of JUnit 5 extensions for AWS. I am it's author. These extensions can be used to inject clients for AWS services provided by tools like LocalStack or any other AWS-compatible API (including the real AWS, of course). Both AWS Java SDK v 2.x and v 1.x are supported. You aws-junit5 to inject clients for S3, DynamoDB, Kinesis, SES, SNS and SQS. Read more in the user guide.

does `aws-junit5` _require_ an external resource like localstack to be running? Or does it provide a way to run everything in memory for the duration of the unit test? — dlipofsky, Feb 08 '22 at 23:01
It doesn't strictly require anything. You could even inject clients for a real AWS. But localstack is IMO the Best option for the tests. — madhead, Feb 09 '22 at 01:26
it sounds like you are saying that you have a choice of which external resource you use, but it does require an external resource. — dlipofsky, Feb 09 '22 at 14:15

How can I unit test the getObject method of the AWS S3 SDK using Java?

6 Answers6

S3Repo to Test

Spock Test