4

I created a cluster according to AWS' tutorials and it supposed to be working just fine but when I try to connect to it, I get the following error

java.io.IOException: failed to configure cluster endpoints from hosts: [mydaxcluster.ximfba.clustercfg.dax.use1.cache.amazonaws.com:8111] at com.amazon.dax.client.cluster.Source$AutoconfSource.pull(Source.java:128) ~[DaxJavaClient-latest.jar:1.0.792.0] at com.amazon.dax.client.cluster.Source$AutoconfSource.update(Source.java:60) ~[DaxJavaClient-latest.jar:1.0.792.0] at com.amazon.dax.client.cluster.Source$AutoconfSource.refresh(Source.java:51) ~[DaxJavaClient-latest.jar:1.0.792.0] at com.amazon.dax.client.cluster.Cluster.refresh(Cluster.java:311) ~[DaxJavaClient-latest.jar:1.0.792.0] at com.amazon.dax.client.cluster.Cluster.refresh(Cluster.java:294) ~[DaxJavaClient-latest.jar:1.0.792.0] at com.amazon.dax.client.cluster.Cluster.startup(Cluster.java:191) ~[DaxJavaClient-latest.jar:1.0.792.0] at com.amazon.dax.client.cluster.Cluster.startup(Cluster.java:143) ~[DaxJavaClient-latest.jar:1.0.792.0] at com.amazon.dax.client.dynamodbv2.ClusterDaxClient.(ClusterDaxClient.java:159) [DaxJavaClient-latest.jar:1.0.792.0] at com.amazon.dax.client.dynamodbv2.ClusterDaxClient.(ClusterDaxClient.java:130) [DaxJavaClient-latest.jar:1.0.792.0] at com.poc.dynamo.controller.DataController.testDax(DataController.java:190) [classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) [spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.11.jar:8.5.11] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) [spring-boot-1.5.1.RELEASE.jar:1.5.1.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:108) [spring-boot-actuator-1.5.1.RELEASE.jar:1.5.1.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) [spring-boot-actuator-1.5.1.RELEASE.jar:1.5.1.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:474) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:783) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:798) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1434) [tomcat-embed-core-8.5.11.jar:8.5.11] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.11.jar:8.5.11] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.11.jar:8.5.11] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] Caused by: java.net.ConnectException: timeout waiting for an available tube at com.amazon.dax.client.SocketTubePool.alloc(SocketTubePool.java:77) ~[DaxJavaClient-latest.jar:1.0.792.0] at com.amazon.dax.client.generated.DaxClientStubs.endpoints_455855874_1(DaxClientStubs.java:449) ~[DaxJavaClient-latest.jar:1.0.792.0] at com.amazon.dax.client.dynamodbv2.DaxClient.endpoints(DaxClient.java:1825) ~[DaxJavaClient-latest.jar:1.0.792.0] at com.amazon.dax.client.cluster.Source$AutoconfSource.pullFrom(Source.java:138) ~[DaxJavaClient-latest.jar:1.0.792.0] at com.amazon.dax.client.cluster.Source$AutoconfSource.pull(Source.java:106) ~[DaxJavaClient-latest.jar:1.0.792.0] ... 75 common frames omitted Suppressed: java.net.ConnectException: timeout waiting for an available tube ... 80 common frames omitted Suppressed: java.net.ConnectException: timeout waiting for an available tube ... 80 common frames omitted

Unable to read item Unable to call cluster endpoint: No endpoints available

TS96
  • 53
  • 1
  • 5

2 Answers2

5

We will need to know more info about how you setup your DAX cluster and client instances to be able to help. Some general ideas to try:

  1. Are you able to connect to the DAX cluster from the EC2 client instance? Try running: nc -z mydaxcluster.ximfba.clustercfg.dax.use1.cache.amazonaws.com 8111
  2. Have you used the default security group for your VPC to create DAX cluster? You will need to authorize inbound traffic on TCP port 8111 to the security group used to create the DAX cluster from the security group of the client instance. Please follow the steps for configuring security group rules for your VPC
Abdelrahman Elhaddad
  • 831
  • 5
  • 7
  • Thanks for the reply, I did all the steps on AWS' website, including configuring the security group rules. I wasn't able to connect to my cluster. – TS96 Jul 05 '17 at 20:39
  • 1
    Can you try the `nc -z` command above from your EC2 client instance and paste the output. If the result is not `Connection to mydaxcluster.ximfba.clustercfg.dax.use1.cache.amazonaws.com 8111 port [tcp/*] succeeded!`. Then we need to troubleshoot your VPC setup. – Abdelrahman Elhaddad Jul 05 '17 at 21:07
  • I'm new to AWS so what do you mean when you say "your EC2 client instance"? Thanks. – TS96 Jul 05 '17 at 21:19
  • 3
    To be able to use DAX, your application (DAX client software) needs to run on an EC2 instance that is in the same VPC used to create DAX cluster. Please refer to the [steps](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DAX.client.launch-ec2-instance.html) in the documentation on how to create an instance on EC2 and how to run DAX client software on that instance. – Abdelrahman Elhaddad Jul 05 '17 at 21:32
  • For some reason I have never seen these particular steps before, the ones I was following are different. I'll get started on these and update you, thanks a lot! – TS96 Jul 06 '17 at 13:18
  • Hey, thanks again for helping. I went through all the steps and created an EC2 instance and the security group I chose should allow connections on port 22. When I try to ssh into my instance I get the following: ssh: connect to host ec2-32-347-197-250.compute-1.amazonaws.com port 22: Connection refused – TS96 Jul 06 '17 at 14:52
  • To be able to ssh to an EC2 instance in a VPC, the instance needs to be in a public subnet. A public subnet is a subnet associated with a route table that has a route to the internet. How to set it up depends on your architecture. You can find examples for different [scenarios](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenarios.html) in the documentation. – Abdelrahman Elhaddad Jul 06 '17 at 16:14
  • I had the same problem and I had to add inbound and outbound rules to allow TCP traffic port 8111 to my Security Group for EC2 host and the DAX Cluster Security Group for the dax hosts - Only then the `nc -z` command worked and then dax also worked. The EC2 Security group and DAX Security groups had to be in the same VPC and you need to specify that they talk to each other – AztecWarrior_25 Oct 05 '17 at 01:20
0

I had faced the same issue, when I tried to connect to a Dax cluster from ECS containers.

Here is the thing that worked for me, as suggested by Abdelrahman Elhaddad

  1. Make sure the subnets and the VPC of the application server and the dax cluster is same.

  2. On your Dax Cluster's security group, ensure inbound TCP traffic is enabled on port 8111 (if your DAX cluster is unencrypted) and 9111 (for encrypted).

  3. And on your server's security group, make sure that TCP outbound requests on 8111/9111 are allowed.

PS: If you are on ECS, modifying docker images can be very cumbersome, use any EC2 instances and try to alter the variables if needed. Also, you can make use of command line network tool to check your security group rules

Adithya
  • 1