79

I'm using DynamoDB local for unit testing. It's not bad, but has some drawbacks. Specifically:

  • You have to somehow start the server before your tests run
  • The server isn't started and stopped before each test so tests become inter-dependent unless you add code to delete all tables, etc. after each test
  • All developers need to have it installed

What I want to do is something like put the DynamoDB local jar, and the other jars upon which it depends, in my test/resources directory (I'm writing in Java). Then before each test I'd start it up, running with -inMemory, and after the test I'd stop it. That way anyone pulling down the git repo gets a copy of everything they need to run the tests and each test is independent of the others.

I have found a way to make this work, but it's ugly, so I'm looking for alternatives. The solution I have is to put a .zip file of the DynamoDB local stuff in test/resources, then in the @Before method, I'd extract it to some temporary directory and start a new java process to execute it. That works, but it's ugly and has some drawbacks:

  • Everyone needs the java executable on their $PATH
  • I have to unpack a zip to the local disk. Using local disk is often dicey for testing, especially with continuous builds and such.
  • I have to spawn a process and wait for it to start for each unit test, and then kill that process after each test. Besides being slow, the potential for left-over processes seems ugly.

It seems like there should be an easier way. DynamoDB Local is, after all, just Java code. Can't I somehow ask the JVM to fork itself and look inside the resources to build a classpath? Or, even better, can't I just call the main method of DynamoDB Local from some other thread so this all happens in a single process? Any ideas?

PS: I am aware of Alternator, but it appears to have other drawbacks so I'm inclined to stick with Amazon's supported solution if I can make it work.

Dmitriy Popov
  • 2,150
  • 3
  • 25
  • 34
Oliver Dain
  • 9,617
  • 3
  • 35
  • 48
  • 2
    As you say that you want to write unit tests - not integration tests - why not use a mock? Something like DynamoDB-mock. This one allows [to be encapsulated as library](http://ddbmock.readthedocs.org/en/latest/pages/getting_started.html#using-ddbmock-for-tests). – cheffe Nov 13 '14 at 10:05
  • 3
    @cheffe, thanks for the thought. That appears to be exactly what I want, but it's Python, not Java so I'd still have to spawn an external executable from my tests just like I'm doing with DynamoDB Local (and make sure all users had the right version of Python installed, had that on their $PATH, etc.). I'm looking for something very much like that, but in Java. Note that creating my own mock would be a huge task since the Dynamo API is pretty rich. – Oliver Dain Nov 13 '14 at 17:57

13 Answers13

87

In order to use DynamoDBLocal you need to follow these steps.

  1. Get Direct DynamoDBLocal Dependency
  2. Get Native SQLite4Java dependencies
  3. Set sqlite4java.library.path to show native libraries

1. Get Direct DynamoDBLocal Dependency

This one is the easy one. You need this repository as explained here.

<!--Dependency:-->
<dependencies>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>DynamoDBLocal</artifactId>
        <version>1.11.0.1</version>
        <scope></scope>
    </dependency>
</dependencies>
<!--Custom repository:-->
<repositories>
    <repository>
        <id>dynamodb-local</id>
        <name>DynamoDB Local Release Repository</name>
        <url>https://s3-us-west-2.amazonaws.com/dynamodb-local/release</url>
    </repository>
</repositories>

2. Get Native SQLite4Java dependencies

If you do not add these dependencies, your tests will fail with 500 internal error.

First, add these dependencies:

<dependency>
    <groupId>com.almworks.sqlite4java</groupId>
    <artifactId>sqlite4java</artifactId>
    <version>1.0.392</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.almworks.sqlite4java</groupId>
    <artifactId>sqlite4java-win32-x86</artifactId>
    <version>1.0.392</version>
    <type>dll</type>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.almworks.sqlite4java</groupId>
    <artifactId>sqlite4java-win32-x64</artifactId>
    <version>1.0.392</version>
    <type>dll</type>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.almworks.sqlite4java</groupId>
    <artifactId>libsqlite4java-osx</artifactId>
    <version>1.0.392</version>
    <type>dylib</type>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.almworks.sqlite4java</groupId>
    <artifactId>libsqlite4java-linux-i386</artifactId>
    <version>1.0.392</version>
    <type>so</type>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.almworks.sqlite4java</groupId>
    <artifactId>libsqlite4java-linux-amd64</artifactId>
    <version>1.0.392</version>
    <type>so</type>
    <scope>test</scope>
</dependency>

Then, add this plugin to get native dependencies to specific folder:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>2.10</version>
            <executions>
                <execution>
                    <id>copy</id>
                    <phase>test-compile</phase>
                    <goals>
                        <goal>copy-dependencies</goal>
                    </goals>
                    <configuration>
                        <includeScope>test</includeScope>
                        <includeTypes>so,dll,dylib</includeTypes>
                        <outputDirectory>${project.basedir}/native-libs</outputDirectory>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

3. Set sqlite4java.library.path to show native libraries

As last step, you need to set sqlite4java.library.path system property to native-libs directory. It is OK to do that just before creating your local server.

System.setProperty("sqlite4java.library.path", "native-libs");

After these steps you can use DynamoDBLocal as you want. Here is a Junit rule that creates local server for that.

import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient;
import com.amazonaws.services.dynamodbv2.local.main.ServerRunner;
import com.amazonaws.services.dynamodbv2.local.server.DynamoDBProxyServer;
import org.junit.rules.ExternalResource;

import java.io.IOException;
import java.net.ServerSocket;

/**
 * Creates a local DynamoDB instance for testing.
 */
public class LocalDynamoDBCreationRule extends ExternalResource {

    private DynamoDBProxyServer server;
    private AmazonDynamoDB amazonDynamoDB;

    public LocalDynamoDBCreationRule() {
        // This one should be copied during test-compile time. If project's basedir does not contains a folder
        // named 'native-libs' please try '$ mvn clean install' from command line first
        System.setProperty("sqlite4java.library.path", "native-libs");
    }

    @Override
    protected void before() throws Throwable {

        try {
            final String port = getAvailablePort();
            this.server = ServerRunner.createServerFromCommandLineArgs(new String[]{"-inMemory", "-port", port});
            server.start();
            amazonDynamoDB = new AmazonDynamoDBClient(new BasicAWSCredentials("access", "secret"));
            amazonDynamoDB.setEndpoint("http://localhost:" + port);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

    @Override
    protected void after() {

        if (server == null) {
            return;
        }

        try {
            server.stop();
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

    public AmazonDynamoDB getAmazonDynamoDB() {
        return amazonDynamoDB;
    }

    private String getAvailablePort() {
        try (final ServerSocket serverSocket = new ServerSocket(0)) {
            return String.valueOf(serverSocket.getLocalPort());
        } catch (IOException e) {
            throw new RuntimeException("Available port was not found", e);
        }
    }
}

You can use this rule like this

@RunWith(JUnit4.class)
public class UserDAOImplTest {

    @ClassRule
    public static final LocalDynamoDBCreationRule dynamoDB = new LocalDynamoDBCreationRule();
}
Mohan Radhakrishnan
  • 3,002
  • 5
  • 28
  • 42
bhdrkn
  • 6,244
  • 5
  • 35
  • 42
  • 9
    I found that the DynamoDBLocal dependency automatically brought in sqlite4java and the extra dependencies didn't need to be specified manually. – Jeffery Grajkowski Aug 22 '16 at 18:08
  • @JefferyGrajkowski I tried that as well but I cannot make it working without native libraries. What is your DDB local version? Maybe they updated dependencies. – bhdrkn Aug 24 '16 at 17:12
  • 2
    I use `com.amazonaws:DynamoDBLocal:1.+`. I figured it was best to stay on latest because the service itself is also going to update whether I like it or not. That works out to 1.11.0 right now. – Jeffery Grajkowski Aug 25 '16 at 22:36
  • 1
    I tried this solution along with suggestions from @JefferyGrajkowski and it worked like a charm. Thanks. – Mingliang Liu Sep 20 '16 at 22:02
  • 5
    Very good answer. I would put the native-libs in target: `${project.basedir}/native-libs` and `System.setProperty("sqlite4java.library.path", "target/native-libs");` – Bart Swennenhuis Sep 11 '18 at 08:27
  • This is a great answer and helped me to write my first test case. Thanks a lot man for your dedication for such detailed answer. – Arefe Jan 22 '22 at 01:53
  • @bhdrkn This answer is very helpful and you are right, I also can't make it run without the native libraries. – Arefe Jan 24 '22 at 14:31
  • The SQLite extension is a transitive dependency of the DynamoDB emulator. You don't need to add it separately. See [my answer](https://stackoverflow.com/a/71722197/2172566) for more info. – forresthopkinsa Apr 03 '22 at 00:52
27

In August 2018 Amazon announced new Docker image with Amazon DynamoDB Local onboard. It does not require downloading and running any JARs as well as adding using third-party OS-specific binaries (I'm talking about sqlite4java).

It is as simple as starting a Docker container before the tests:

docker run -p 8000:8000 amazon/dynamodb-local

You can do that manually for local development, as described above, or use it in your CI pipeline. Many CI services provide an ability to start additional containers during the pipeline that can provide dependencies for your tests. Here is an example for Gitlab CI/CD:

test:
  stage: test
  image: openjdk:8-alpine
  services:
    - name: amazon/dynamodb-local
      alias: dynamodb-local
  script:
    - DYNAMODB_LOCAL_URL=http://dynamodb-local:8000 ./gradlew clean test

Or Bitbucket Pipelines:

definitions:
  services:
    dynamodb-local:
      image: amazon/dynamodb-local
…
step:
  name: test
  image:
    name: openjdk:8-alpine
  services:
    - dynamodb-local
  script:
    - DYNAMODB_LOCAL_URL=http://localhost:8000 ./gradlew clean test

And so on. The idea is to move all the configuration you can see in other answers out of your build tool and provide the dependency externally. Think of it as of dependency injection / IoC but for the whole service, not just a single bean.

After you've started the container you can create a client pointing to it:

private AmazonDynamoDB createAmazonDynamoDB(final DynamoDBLocal configuration) {
    return AmazonDynamoDBClientBuilder
        .standard()
        .withEndpointConfiguration(
            new AwsClientBuilder.EndpointConfiguration(
                "http://localhost:8000",
                Regions.US_EAST_1.getName()
            )
        )
        .withCredentials(
            new AWSStaticCredentialsProvider(
                // DynamoDB Local works with any non-null credentials
                new BasicAWSCredentials("", "")
            )
        )
        .build();
}

Now to the original questions:

You have to somehow start the server before your tests run

You can just start it manually, or prepare a developsers' script for it. IDEs usually provide a way to run arbitrary commands before executing a task, so you can make IDE to start the container for you. I think that running something locally should not be a top priority in this case, but instead you should focus on configuring CI and let the developers start the container as it's comfortable to them.

The server isn't started and stopped before each test so tests become inter-dependent unless you add code to delete all tables, etc. after each test

That's trueee, but… You should not start and stop such heavyweight things and recreate tables before / after each test. DB tests are almost always inter-dependent and that's ok for them. Just use unique values for each test case (e.g. set item's hash key to ticket id / specific test case id you're working on). As for the seed data, I'd recommend moving it from the build tool and test code as well. Either make your own image with all the data you need or use AWS CLI to create tables and insert data. Follow the single responsibility principle and dependency injection principles: your test code must not do anything but tests. All the environment (tables and data in this case should be provided for them). Creating a table in a test is wrong, because in a real life that table already exist (unless you're testing a method that actually creates a table, of course).

All developers need to have it installed

Docker should be a must for every developer in 2018, so that's not a problem.


And if you're using JUnit 5, it can be a good idea to use a DynamoDB Local extension that will inject the client in your tests (yes, I'm doing a self-promotion):

  1. Add a dependency on me.madhead.aws-junit5:dynamo-v1

    pom.xml:

    <dependency>
        <groupId>me.madhead.aws-junit5</groupId>
        <artifactId>dynamo-v1</artifactId>
        <version>6.0.1</version>
        <scope>test</scope>
    </dependency>
    

    build.gradle

    dependencies {
        testImplementation("me.madhead.aws-junit5:dynamo-v1:6.0.1")
    }
    
  2. Use the extension in your tests:

    @ExtendWith(DynamoDBLocalExtension.class)
    class MultipleInjectionsTest {
        @DynamoDBLocal(
            url = "http://dynamodb-local-1:8000"
        )
        private AmazonDynamoDB first;
    
        @DynamoDBLocal(
            urlEnvironmentVariable = "DYNAMODB_LOCAL_URL"
        )
        private AmazonDynamoDB second;
    
        @Test
        void test() {
            first.listTables();
            second.listTables();
        }
    }
    
Freewind
  • 193,756
  • 157
  • 432
  • 708
madhead
  • 31,729
  • 16
  • 153
  • 201
  • How would this work for local testing? I shouldn't have to force every poor developer to install docker and have the DynamoDBLocal image running and meticulously configured. – zalpha314 Jul 31 '19 at 21:04
  • 3
    Indeed, a developer who's afraid of Docker in 2019 is a poor developer. You can still use this approach for CI/CD, where everything happens in Docker anyway (most modern CI servers are Docker based, even Jenkins works with Docker). My point is that you don't pollute your test codebase with initiation code, but just provide a service (DynamoDB) externally. It can be a Docker container. Or it can be a `DynamoDBLocal.jar`. Or you can run a [localstack](https://github.com/localstack/localstack). All your tests need to know is the URL in all the cases. – madhead Jul 31 '19 at 23:16
  • 1
    That is a great answer, thanks a lot. I tried to run docker with `docker run` right in `-script` of `gitlab-ci.yml` and had epic problem with connecting to this docker (see https://stackoverflow.com/questions/60326823/gitlab-ci-how-to-connect-to-the-docker-container-started-in-gitlab-ci-yml-scri). Your nice solution with `-services` worked like a charm. – Dmitriy Popov Feb 21 '20 at 09:33
  • Could you also be so kind to describe how to pass command line arguments `-jar DynamoDBLocal.jar -sharedDb` on Gitlab CI to make the database shared? – Dmitriy Popov Feb 24 '20 at 11:59
  • I tried `command: [ "-jar DynamoDBLocal.jar -sharedDb" ]` and `command: [ "-sharedDb", "" ]`, but Gitlab CI says `2020-02-24T12:11:55.675930668Z Unrecognized option: -sharedDb 2020-02-24T12:11:55.675989051Z Error: Could not create the Java Virtual Machine. 2020-02-24T12:11:55.675994696Z Error: A fatal exception has occurred. Program will exit.` Also tried `command: [ "docker run -p 8000:8000 -v $(pwd)/local/dynamodb:/data/ amazon/dynamodb-local -jar DynamoDBLocal.jar -sharedDb" ]` with no success. The only relevant link I found is https://forums.aws.amazon.com/thread.jspa?messageID=866259. – Dmitriy Popov Feb 24 '20 at 12:17
  • @DmitriyPopov, try to split the string: `command: ["-jar", "DynamoDBLocal.jar", "-sharedDb"]` – madhead Feb 24 '20 at 12:53
  • 2
    @madhead On the contrary, when your CI/CD having your build depend on running sub-containers is a massive pain. Running docker-in-docker is rife with problems. – Magnus Jun 03 '20 at 23:09
  • @Magnus, well, some of the CI/CD providers (I can say for GitLab CI/CD and BitBucket Pipelines) have a notion of "services". It's a container that the CI/CD spins in a parallel to your job and binds to you job container. GitHub Action [has them as well](https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idservices), but I've never tested them in practice, it just seems like a similar approach. – madhead Jun 03 '20 at 23:19
  • And for local development, you can use localstack or Testcontainers. The main idea I am preaching here is decoupling the code from the backing services and treating them as a replacable detachments, like in [12factor](https://12factor.net/backing-services) principles. – madhead Jun 03 '20 at 23:22
  • So instead of using Gradle tasks to spin up the database, or run it in `@Before` method in JUnit, or doing it in any other (imperative) way, I just propose to declare a single dependency on the database url in the code and provide the database (resource) externally. – madhead Jun 03 '20 at 23:26
  • Moreover, that `com.amazonaws:DynamoDBLocal` artifact from the other answers has a pretty tricky setup with some native binaries and library pathes. So, I would argue that running a Docker container is more complex then hassling with some `.so` and `.libs`. – madhead Jun 03 '20 at 23:30
  • 1
    Spinning up docker images for tests is slow. Alternatively, if you rely on external resources to be set up and maintained manually, your tests become fragile and pain for devs to run. – Adrian Baker Aug 29 '21 at 20:19
  • @AdrianBaker, yes, and? How do you test your code working with DynamoDB then? – madhead Aug 30 '21 at 07:57
  • 1
    Using JVM embedded instances of dynamodb. – Adrian Baker Aug 30 '21 at 21:22
  • And how do we test then non-JVM technologies? What if the app needs a RabbitMQ? – madhead Aug 30 '21 at 22:51
  • Even the DynamoDB is not a 100% JVM technology. Good luck with those native binaries from other answers. – madhead Aug 30 '21 at 22:52
  • 1
    All the dynamodb-local docker image does is run the same jar (https://hub.docker.com/layers/amazon/dynamodb-local/1.16.0/images/sha256-0c8f11e69ccf895c7ec97a7c9991d4654971f0e5e3ae42292de1c514f3e03cfa?context=explore). Docker isn't always the only or best approach. More like just the lowest common denominator. – Adrian Baker Sep 02 '21 at 17:10
  • @AdrianBaker, but first, take a deeper look at this. It's not just a JAR. It's a TAR file with a lots of things, and one of those things — check it if you don't trust me — is the whole SQLite. I mean, the SQLIte binaries and JARs. And, you know, binaries are platform dependend. Go and read the other answers here, they are just about that — getting this SQLIte thing work. – madhead Sep 02 '21 at 21:11
  • …аnd this is kind of hassle. So my answer is just about thatt: there are different ways to accomplish things. I'm not saying Docker is the best, but it's definitely worst trying and it is suitable in many scenarios. If you're not happy with Docker or think it makes tests fragile — just skip it instead of denying. – madhead Sep 02 '21 at 21:15
  • 1
    The tar is just https://mvnrepository.com/artifact/com.amazonaws/DynamoDBLocal/1.15.0 plus the runtime dependencies. So the exact same binaries you get when you add it to the classpath. – Adrian Baker Sep 02 '21 at 22:26
  • Which contains platform binaries. Do `ls -la /home/dynamodblocal/DynamoDBLocal_lib/lib*` in the container and see. `/home/dynamodblocal` is the directory it is unpacked. – madhead Sep 02 '21 at 22:30
  • Those are exactly the same platform binaries linked as runtime maven dependencies in https://s3-us-west-2.amazonaws.com/dynamodb-local/release/com/amazonaws/DynamoDBLocal/1.16.0/DynamoDBLocal-1.16.0.pom, see the "so" and "dll" runtime dependencies in that pom.xml . There is no extra "magic" in the docker image. – Adrian Baker Sep 02 '21 at 22:48
  • But I never stated that there is any magic in that Docker image, did I? I'm just saying that Docker image already contains everything inside and it is 100% working with one simple command (`docker start`) or a simple config (Testcontainers or CI/CD service). If you already have Docker or run your CI in a modern environment like GitHub Actions — use it. If you have some issues with Docker or just don't like it — don't use it. That's simple. – madhead Sep 02 '21 at 23:28
  • I just don't understand you, commentors, commenting this post with "Docker is an overkill here" / "Docker is too complex" / "Docker makes tests fragile". Well, it is complex, it may be an overkill and it is an external dependency to you tests, indeed (just like any other JARs). Skip the answer, it's not for you. Let it be here for those bravehearts who is ok to use Docker! – madhead Sep 02 '21 at 23:31
20

This is a restating of bhdrkn's answer for Gradle users (his is based on Maven). It's still the same three steps:

  1. Get Direct DynamoDBLocal Dependency
  2. Get Native SQLite4Java dependencies
  3. Set sqlite4java.library.path to show native libraries

1. Get Direct DynamoDBLocal Dependency

Add to the dependencies section of your build.gradle file...

dependencies {
    testCompile "com.amazonaws:DynamoDBLocal:1.+"
}

2. Get Native SQLite4Java dependencies

The sqlite4java libraries will already be downloaded as a dependency of DynamoDBLocal, but the library files need to be copied to the right place. Add to your build.gradle file...

task copyNativeDeps(type: Copy) {
    from(configurations.compile + configurations.testCompile) {
        include '*.dll'
        include '*.dylib'
        include '*.so'
    }
    into 'build/libs'
}

3. Set sqlite4java.library.path to show native libraries

We need to tell Gradle to run copyNativeDeps for testing and tell sqlite4java where to find the files. Add to your build.gradle file...

test {
    dependsOn copyNativeDeps
    systemProperty "java.library.path", 'build/libs'
}
mkobit
  • 43,979
  • 12
  • 156
  • 150
Jeffery Grajkowski
  • 3,982
  • 20
  • 23
  • @Jeffery I am getting the below errors: testMapRtbUser STANDARD_ERROR 17:39:41.931 [DEBUG] [TestEventLogger] 2017-08-29 17:39:41.929:WARN:oejs.AbstractHttpConnection:/ 17:39:41.931 [DEBUG] [TestEventLogger] java.lang.NoSuchMethodError: com.amazon.dynamodb.grammar.DynamoDbExpressionParser.parseAttributeValuesMapKeys(Ljava/lang/String;Lorg/antlr/v4/runtime/ANTLRErrorListener;)V However, for the same test if I run it from Eclipse as a Junit tests it runs fine. It only when run by gradle as a test it fails. Later on this times out to be a timeout error the update operation. Help! – Roy Aug 30 '17 at 00:53
  • It sounds like a runtime classpath issue. That class and that method with that signature definitely exists in the latest version of the JAR. Try clearing everything that Gradle has cached and try again. – Jeffery Grajkowski Aug 31 '17 at 19:48
  • I followed this but I keep getting java.lang.RuntimeException: com.amazonaws.SdkClientException: Unable to execute HTTP request: The target server failed to respond when I run my tests. Any idea what am I doing wrong? – Red May 05 '18 at 01:47
  • Have you managed to run and write user tests for DynamoDBLocal when starting it manually? Try writing the manual, easy thing first before automating. – Jeffery Grajkowski May 07 '18 at 21:04
  • 1
    To run tests inside IntelliJ, add `-Djava.library.path=build/libs` to the "VM options" in Run/Debug configuration. – Simon Forsberg Nov 28 '19 at 17:44
  • You don't need to copy the files around. They're already on your system, you just need to instruct the JVM where to find them. See [my answer](https://stackoverflow.com/a/71722197/2172566) for more info. – forresthopkinsa Apr 03 '22 at 00:53
19

You can use DynamoDB Local as a Maven test dependency in your test code, as is shown in this announcement. You can run over HTTP:

import com.amazonaws.services.dynamodbv2.local.main.ServerRunner;
import com.amazonaws.services.dynamodbv2.local.server.DynamoDBProxyServer;

final String[] localArgs = { "-inMemory" };
DynamoDBProxyServer server = ServerRunner.createServerFromCommandLineArgs(localArgs);
server.start();
AmazonDynamoDB dynamodb = new AmazonDynamoDBClient();
dynamodb.setEndpoint("http://localhost:8000");
dynamodb.listTables();
server.stop();

You can also run in embedded mode:

import com.amazonaws.services.dynamodbv2.local.embedded.DynamoDBEmbedded;

AmazonDynamoDB dynamodb = DynamoDBEmbedded.create();
dynamodb.listTables();
Alexander Patrikalakis
  • 5,054
  • 1
  • 30
  • 48
  • 1
    For the first option using the ServerRunner it starts ok but as soon as I try to create a table, I get `AmazonServiceException The request processing has failed because of an unknown error, exception or failure. (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: InternalFailure; Request ID: ea0eff34-65e4-49d5-8ae9-3bfbfec9136e)` – leonardoborges Aug 04 '15 at 01:50
  • 7
    For the Embedded version I get a `NullPointerException` from SQLLite in the `initializeMetadataTables` method. :( – leonardoborges Aug 04 '15 at 01:51
  • Make sure you provide the full path to sqlite4java JNI libraries as part of the -Dsqlite4java.library.path=/the/path/to/sqlite/for/java/jni/libraries system property. – Alexander Patrikalakis Aug 04 '15 at 01:52
  • Is that necessary when using the inMemory option? – leonardoborges Aug 04 '15 at 02:11
  • DynamoDB Local requires sqlite4java libraries regardless of whether you use the inMemory option or not. – Alexander Patrikalakis Aug 04 '15 at 02:12
  • 4
    There doesn't seem to be any information about that in the official [repository containing example code](https://github.com/awslabs/aws-dynamodb-examples/blob/master/src/test/java/com/amazonaws/services/dynamodbv2/DynamoDBLocalFixture.java). – leonardoborges Aug 04 '15 at 02:18
  • Actually, this information is in the official repository containing example code, in the start-dynamodb-local profile of the test target. I will add a target that runs this class as a test to make this clear. https://github.com/awslabs/aws-dynamodb-examples/blob/master/pom.xml#L183 – Alexander Patrikalakis Aug 04 '15 at 02:21
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/85053/discussion-between-leonardoborges-and-alexander-patrikalakis). – leonardoborges Aug 04 '15 at 02:27
  • I tried the solutions provided today, however it seems like the repo at http://dynamodb-local.s3-website-us-west-2.amazonaws.com/release doesn't exist (or premissions are set to deny any access) and thus I can't download an artifact described in https://forums.aws.amazon.com/ann.jspa?annID=3148 – Daniel Gruszczyk Sep 29 '15 at 13:07
  • I did `rm -rf ~/.m2/repository/com/amazonaws/DynamoDBLocal/` and then `mvn install` and then `mvn test -Pstart-dynamodb-local` and I was able to navigate to the DynamoDB shell at http://localhost:8000/shell/. The issue seems to be resolved, can you confirm? – Alexander Patrikalakis Sep 29 '15 at 13:43
  • @AlexanderPatrikalakis Hi Alexander, the problem turned out to be my maven config. It indeed works now. thanks for your help. – Daniel Gruszczyk Sep 30 '15 at 09:29
6

I have wrapped the answers above into two JUnit rules that does not require changes to the build script as the rules handles the native library stuff. I did this as I found that Idea did not like the Gradle/Maven solutions as it just went off and did its own thing anyhoos.

This means the steps are:

  • Get the AssortmentOfJUnitRules version 1.5.32 or above dependency
  • Get the Direct DynamoDBLocal dependency
  • Add the LocalDynamoDbRule or HttpDynamoDbRule to your JUnit test.

Maven:

<!--Dependency:-->
<dependencies>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>DynamoDBLocal</artifactId>
        <version>1.11.0.1</version>
        <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.github.mlk</groupId>
      <artifactId>assortmentofjunitrules</artifactId>
      <version>1.5.36</version>
      <scope>test</scope>
    </dependency>
</dependencies>
<!--Custom repository:-->
<repositories>
    <repository>
        <id>dynamodb-local</id>
        <name>DynamoDB Local Release Repository</name>
        <url>https://s3-us-west-2.amazonaws.com/dynamodb-local/release</url>
    </repository>
</repositories>

Gradle:

repositories {
  mavenCentral()

   maven {
    url = "https://s3-us-west-2.amazonaws.com/dynamodb-local/release"
  }
}

dependencies {
    testCompile "com.github.mlk:assortmentofjunitrules:1.5.36"
    testCompile "com.amazonaws:DynamoDBLocal:1.+"
}

Code:

public class LocalDynamoDbRuleTest {
  @Rule
  public LocalDynamoDbRule ddb = new LocalDynamoDbRule();

  @Test
  public void test() {
    doDynamoStuff(ddb.getClient());
  }
}
mkobit
  • 43,979
  • 12
  • 156
  • 150
Michael Lloyd Lee mlk
  • 14,561
  • 3
  • 44
  • 81
  • I ran into an issue with this. I get a `NPE` in `LocalDynamoDbRule.java:38` (`DynamoDBEmbedded.create()` returns `null)`. Before that, the native SQLite part logs: `com.almworks.sqlite4java.SQLiteException: [-91] cannot load library: com.almworks.sqlite4java.SQLiteException: [-91] sqlite4java cannot find native library`. I use `com.github.mlk:DynamoDBLocal:1.11.119` and `com.github.mlk:assortmentofjunitrules:1.5.39`. Shouldn't the `LocalDynamoDbRule` take care of all the native SQLite stuff? – scho Jul 16 '18 at 17:04
  • [Here](https://gist.github.com/scho/3101c83488e61d32234ce79c5db6ac06) is the complete stacktrace of the error I get. – scho Jul 16 '18 at 17:16
  • It should be handling the native library stuff for you yes. I'll look into it. Thanks – Michael Lloyd Lee mlk Jul 17 '18 at 18:08
  • Could you send me a snipped of your POM/gradle/whatever as well please? Thanks. – Michael Lloyd Lee mlk Jul 18 '18 at 09:41
  • I've been able to replicate with a minimal POM. It appears this is an issue with Maven that does not effect Gradle. I will continue to investigate. https://github.com/mlk/AssortmentOfJUnitRules/issues/2 – Michael Lloyd Lee mlk Jul 18 '18 at 13:32
5

Try out tempest-testing! It ships a JUnit4 Rule and a JUnit5 Extension. It also supports both AWS SDK v1 and SDK v2.

Tempest provides a library for testing DynamoDB clients using DynamoDBLocal . It comes with two implementations:

  • JVM: This is the preferred option, running a DynamoDBProxyServer backed by sqlite4java, which is available on most platforms.
  • Docker: This runs dynamodb-local in a Docker container.

Feature matrix:

Feature tempest-testing-jvm tempest-testing-docker
Start up time ~1s ~10s
Memory usage Less More
Dependency sqlite4java native library Docker

To use tempest-testing, first add this library as a test dependency:

For AWS SDK 1.x:

dependencies {
  testImplementation "app.cash.tempest:tempest-testing-jvm:1.5.2"
  testImplementation "app.cash.tempest:tempest-testing-junit5:1.5.2"
}
// Or
dependencies {
  testImplementation "app.cash.tempest:tempest-testing-docker:1.5.2"
  testImplementation "app.cash.tempest:tempest-testing-junit5:1.5.2"
}

For AWS SDK 2.x:

dependencies {
  testImplementation "app.cash.tempest:tempest2-testing-jvm:1.5.2"
  testImplementation "app.cash.tempest:tempest2-testing-junit5:1.5.2"
}
// Or
dependencies {
  testImplementation "app.cash.tempest:tempest2-testing-docker:1.5.2"
  testImplementation "app.cash.tempest:tempest2-testing-junit5:1.5.2"
}

Then in tests annotated with @org.junit.jupiter.api.Test, you may add TestDynamoDb as a test extension. This extension spins up a DynamoDB server. It shares the server across tests and keeps it running until the process exits. It also manages test tables for you, recreating them before each test.

class MyTest {
  @RegisterExtension
  TestDynamoDb db = new TestDynamoDb.Builder(JvmDynamoDbServer.Factory.INSTANCE) // or DockerDynamoDbServer
      // `MusicItem` is annotated with `@DynamoDBTable`. Tempest recreates this table before each test.
      .addTable(TestTable.create(MusicItem.TABLE_NAME, MusicItem.class))
      .build();

  @Test
  public void test() {
    PutItemRequest request = // ...;
    // Talk to the local DynamoDB.
    db.dynamoDb().putItem(request);
  }

}
Zhixuan Lai
  • 51
  • 1
  • 2
  • I migrated some existing container-based test code over to the JVM mode of this library, worked well and gave a good speed bump. – Adrian Baker Aug 29 '21 at 20:22
2

It seems like there should be an easier way. DynamoDB Local is, after all, just Java code. Can't I somehow ask the JVM to fork itself and look inside the resources to build a classpath?

You can do something along these lines, but much simpler: programmatically search the classpath for the location of the native libraries, then set the sqlite4java.library.path property before starting DynamoDB. This is the approach implemented in tempest-testing, as well as in this answer (code here) which is why they just work as pure library/classpath dependency and nothing more.

In my case needed access to DynamoDB outside of a JUnit extension, but I still wanted something self-contained in library code, so I extracted the approach it takes:

import com.amazonaws.services.dynamodbv2.local.embedded.DynamoDBEmbedded;
import com.amazonaws.services.dynamodbv2.local.shared.access.AmazonDynamoDBLocal;
import com.google.common.collect.MoreCollectors;
import java.io.File;
import java.util.Arrays;
import java.util.stream.Stream;
import org.junit.jupiter.api.condition.OS;

... 

  public AmazonDynamoDBLocal embeddedDynamoDb() {
    final OS os = Stream.of(OS.values()).filter(OS::isCurrentOs)
        .collect(MoreCollectors.onlyElement());
    final String prefix;
    switch (os) {
      case LINUX:
        prefix = "libsqlite4java-linux-amd64-";
        break;
      case MAC:
        prefix = "libsqlite4java-osx-";
        break;
      case WINDOWS:
        prefix = "sqlite4java-win32-x64-";
        break;
      default:
        throw new UnsupportedOperationException(os.toString());
    }
  
    System.setProperty("sqlite4java.library.path",
        Arrays.asList(System.getProperty("java.class.path").split(File.pathSeparator))
            .stream()
            .map(File::new)
            .filter(file -> file.getName().startsWith(prefix))
            .collect(MoreCollectors.onlyElement())
            .getParent());
    return DynamoDBEmbedded.create();
  }

Not had a chance to test on a lot of platforms, and the error handling could likely be improved.

It's a pity AWS haven't taken the time to make the library more friendly, as this could easily be done in the library code itself.

Adrian Baker
  • 9,297
  • 1
  • 26
  • 22
1

For unit testing at work I use Mockito, then just mock the AmazonDynamoDBClient. then mock out the returns using when. like the following:

when(mockAmazonDynamoDBClient.getItem(isA(GetItemRequest.class))).thenAnswer(new Answer<GetItemResult>() {
        @Override
        public GetItemResult answer(InvocationOnMock invocation) throws Throwable {
            GetItemResult result = new GetItemResult();
            result.setItem( testResultItem );
            return result;
        }
    });

not sure if that is what your looking for but that's how we do it.

  • 5
    Thanks for the thought. Mocks are OK, but it can be hard to get the protocol exactly right. So you end up testing if you code works assuming Dynamo (or whatever) behaves the way you think it behaves (the way you mocked it), but you're not testing if you code actually works with Dynamo. If you're wrong about how Dynamo works, your code and tests make the same assumptions so things pass but you have bugs. – Oliver Dain Mar 04 '15 at 20:59
  • 3
    It sounds like your doing integration tests, for that you shouldn't have to many tests. you just want to make sure you can do the basic operations. Basically verifying that you have things connected correctly. In the past what I will do is spin up a local instance in the test. then you would have hard coded values you save, read, and delete from your local database. Other than that what are these tests of your doing? I always recommend having unit tests (test's just one thing the rest I mock) and integration tests (everything real) – Steve Smith Mar 05 '15 at 23:11
  • 3
    Theoretically mocking is the right way for unit test, but local DDB can make sure the code is right in a more promising way. – Cherish Feb 17 '16 at 17:39
1

Shortest solution with fix for sqlite4java.SQLiteException UnsatisfiedLinkError if it is a java/kotlin project built with gradle (a changed $PATH is not needed).

repositories {
    // ... other dependencies
    maven { url 'https://s3-us-west-2.amazonaws.com/dynamodb-local/release' } 
}

dependencies {
    testImplementation("com.amazonaws:DynamoDBLocal:1.13.6")
}

import org.gradle.internal.os.OperatingSystem
test {
    doFirst {
        // Fix for: UnsatisfiedLinkError -> provide a valid native lib path
        String nativePrefix = OperatingSystem.current().nativePrefix
        File nativeLib = sourceSets.test.runtimeClasspath.files.find {it.name.startsWith("libsqlite4java") && it.name.contains(nativePrefix) } as File
        systemProperty "sqlite4java.library.path", nativeLib.parent
    }
}

Straightforward usage in test classes (src/test):

private lateinit var db: AmazonDynamoDBLocal

@BeforeAll
fun runDb() { db = DynamoDBEmbedded.create() }

@AfterAll
fun shutdownDb() { db.shutdown() }
user1185087
  • 4,468
  • 1
  • 30
  • 38
0

There are couple of node.js wrappers for DynamoDB Local. These allows to easily execute unit tests combining with task runners like gulp or grunt. Try dynamodb-localhost, dynamodb-local

Ashan
  • 18,898
  • 4
  • 47
  • 67
0

I have found that the amazon repo as no index file, so does not seem to function in a way that allows you to bring it in like this:

maven {
   url = "https://s3-us-west-2.amazonaws.com/dynamodb-local/release"
}

The only way I could get the dependencies to load is by downloading DynamoDbLocal as a jar and bringing it into my build script like this:

dependencies {
    ...
    runtime files('libs/DynamoDBLocal.jar')
    ...
}

Of course this means that all the SQLite and Jetty dependencies need to be brought in by hand - I'm still trying to get this right. If anyone knows of a reliable repo for DynamoDbLocal, I would really love to know.

Michael Coxon
  • 3,337
  • 8
  • 46
  • 68
0

You could also use this lightweight test container 'Dynalite'

https://www.testcontainers.org/modules/databases/dynalite/

From testcontainers:

Dynalite is a clone of DynamoDB, enabling local testing. It's light and quick to run.

baitmbarek
  • 2,440
  • 4
  • 18
  • 26
0

The DynamoDB Gradle dependency already includes the SQLite libraries. You can pretty easily instruct the Java runtime to use it in your Gradle build script. Here's my build.gradle.kts as an example:

import org.apache.tools.ant.taskdefs.condition.Os

plugins {
    application
}

repositories {
    mavenCentral()
    maven {
        url = uri("https://s3-us-west-2.amazonaws.com/dynamodb-local/release")
    }
}

dependencies {
    implementation("com.amazonaws:DynamoDBLocal:[1.12,2.0)")
}

fun getSqlitePath(): String? {
    val dirName = when {
        Os.isFamily(Os.FAMILY_MAC) -> "libsqlite4java-osx"
        Os.isFamily(Os.FAMILY_UNIX) -> "libsqlite4java-linux-amd64"
        Os.isFamily(Os.FAMILY_WINDOWS) -> "sqlite4java-win32-x64"
        else -> throw kotlin.Exception("DynamoDB emulator cannot run on this platform")
    }
    return project.configurations.runtimeClasspath.get().find { it.name.contains(dirName) }?.parent
}

application {
    mainClass.set("com.amazonaws.services.dynamodbv2.local.main.ServerRunner")
    applicationDefaultJvmArgs = listOf("-Djava.library.path=${getSqlitePath()}")
}

tasks.named<JavaExec>("run") {
    args("-inMemory")
}

forresthopkinsa
  • 1,339
  • 1
  • 24
  • 29