Scenario:
We have a streaming Java application that is performing read/write requests to DynamoDB via DAX with the most recent DAX client instance. When performing both simple integration tests and performance testing we intermittently see exceptions about no endpoints being available.
com.amazonaws.AmazonClientException: Unable to call cluster endpoint: No endpoints available
at com.amazon.dax.client.dynamodbv2.ClusterDaxClient$ClusterReadRetryHandler.check(ClusterDaxClient.java:699)
at com.amazon.dax.client.dynamodbv2.ClusterDaxClient.invoke(ClusterDaxClient.java:649)
....
IOException: No endpoints available
at com.amazon.dax.client.cluster.Cluster.readClient(Cluster.java:680)
at com.amazon.dax.client.dynamodbv2.ClusterDaxClient$2.getClient(ClusterDaxClient.java:138)
at com.amazon.dax.client.dynamodbv2.ClusterDaxClient.invoke(ClusterDaxClient.java:631)
... 23 more
To clarify, intermittent meaning that some requests will be successful, then some will fail, only to have the requests once again be successful all within a single test (integration or performance). In fact, the majority of these test runs will be successful w/out ever seeing this error but it does occasionally happen as described in the last sentence.
The dev instance of the DAX and DynamoDB are standing instances, meaning that running these tests does not re-infrastructure/deploy either the DAX or DynamoDB instances for a clean slate (they are left up and running between test runs).
Question:
Has anyone else seen a similar pattern of this intermittent behavior at small scale tests (e.g. integration/e2e testing) or larger scale testing (e.g. performance testing) with DAX? I know that this has been discussed on other tickets for scenarios where people cannot ever connect to DAX, but there has been no issues or scenarios that I have seen identifying intermittent behavior like this.
