0

I was following this tutorial on the Mkyong site https://www.mkyong.com/spring-boot/spring-boot-spring-data-elasticsearch-example/

It uses the spring-boot-starter-data-elasticsearch library to connect to elasticSearch

I was using the following details to try and connect to an elastic search instance on Amazon`

elasticsearch.host=<search address in amazon elastic search console>
elasticsearch.port=9200
elasticsearch.clustername=<accountId:clusterName>`

However I keep getting the following errors

 2017-05-16 18:37:24.308  INFO 7108 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Overriding bean definition for bean 'elasticsearchTemplate' with a different definition: replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=esConfig; factoryMethodName=elasticsearchTemplate; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [com/mkyong/EsConfig.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.data.elasticsearch.ElasticsearchDataAutoConfiguration; factoryMethodName=elasticsearchTemplate; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/data/elasticsearch/ElasticsearchDataAutoConfiguration.class]]
2017-05-16 18:37:24.859  INFO 7108 --- [           main] org.elasticsearch.plugins                : [Rawhide Kid] modules [], plugins [], sites []
2017-05-16 18:37:46.911  INFO 7108 --- [           main] org.elasticsearch.client.transport       : [Rawhide Kid] failed to connect to node [{#transport#-1}{55.66.223.158}{55.66.223.158:9200}], removed from nodes list

org.elasticsearch.transport.ConnectTransportException: [][55.66.223.158:9200] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:967) ~[elasticsearch-2.4.4.jar:2.4.4]
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:933) ~[elasticsearch-2.4.4.jar:2.4.4]
    at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:906) ~[elasticsearch-2.4.4.jar:2.4.4]
    at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267) ~[elasticsearch-2.4.4.jar:2.4.4]
    at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:390) ~[elasticsearch-2.4.4.jar:2.4.4]
    at org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:336) [elasticsearch-2.4.4.jar:2.4.4]
    at org.elasticsearch.client.transport.TransportClientNodesService.addTransportAddresses(TransportClientNodesService.java:187) [elasticsearch-2.4.4.jar:2.4.4]
    at org.elasticsearch.client.transport.TransportClient.addTransportAddress(TransportClient.java:243) [elasticsearch-2.4.4.jar:2.4.4]
    at com.mkyong.EsConfig.client(EsConfig.java:45) [classes/:na]
    at com.mkyong.EsConfig$$EnhancerBySpringCGLIB$$7e0ccfee.CGLIB$client$0(<generated>) [classes/:na]
    at com.mkyong.EsConfig$$EnhancerBySpringCGLIB$$7e0ccfee$$FastClassBySpringCGLIB$$83a2b819.invoke(<generated>) [classes/:na]
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) [spring-core-4.3.6.RELEASE.jar:4.3.6.RELEASE]
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:356) [spring-context-4.3.6.RELEASE.jar:4.3.6.RELEASE]
    at com.mkyong.EsConfig$$EnhancerBySpringCGLIB$$7e0ccfee.client(<generated>) [classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_65]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_65]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_65]
    at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_65]
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instant

Any ideas what I need to do to connect to an elastic search instance on Amazon?

Thanks Damien

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Damien Gallagher
  • 535
  • 1
  • 6
  • 16
  • 1
    can you call elasticsearch endpoint from local `curl -XGET 55.66.223.158:9200` from you local or not? – prayagupa May 16 '17 at 17:48
  • I can curl to port 80 but not port 9200 or port 9300 – Damien Gallagher May 16 '17 at 17:57
  • 1
    Does port 80 return elasticsearch info or simply http response? I mean what is the response of `curl -XGET 55.66.223.158:80`. I just wanted to make sure that elasticsearch on your remote machine runs on port 9200, and AWS has firewall open for that port. – prayagupa May 16 '17 at 17:59
  • This is the response of curl on port 80 - { "name" : "Mimir", "cluster_name" : "", "version" : { "number" : "2.3.2", "build_hash" : "72aa8010df1a4fc849da359c9c58acba6c4d9518", "build_timestamp" : "2016-11-14T15:59:50Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search" } – Damien Gallagher May 16 '17 at 18:13
  • 1
    so its running on port 80, but you are connecting to 9200. change your config, `elasticsearch.port=80`. should work – prayagupa May 16 '17 at 18:24
  • that gives me this error - java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50) – Damien Gallagher May 16 '17 at 18:29
  • There are few reasons why `StreamCorruptedException` happens. Can you check your java running on elasticsearch server and your local machine are same? - https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-3-41-4d-52/25247/3 – prayagupa May 16 '17 at 19:04
  • The other reason would be port `80` is be equivalent to `9200` which is default port for http communication for elasticsearch. But there used to be port `9300` for TCP connection from api like Java. I am not what sure what port Amazon runs for tcp transportation. Please check AWS elasticsearch config for that - http://elasticsearch-users.115913.n3.nabble.com/Failing-starting-due-to-transport-layer-exception-td4033399.html – prayagupa May 16 '17 at 19:08
  • Nevermind it would be port 80 for TCP communication as well, as this [answer mentions](http://stackoverflow.com/a/33558330/432903). so check java versions are similar in server and client, BUT also make sure your clustername is right – prayagupa May 16 '17 at 19:11

3 Answers3

0

AWS Elasticsearch doesn't support TCP connections, only HTTP connections. This explains why your CURL commands work, but your java connection doesn't work. 

TCP transport
The service supports HTTP on port 80, but does not support TCP transport.

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-supported-resources.html

https://forums.aws.amazon.com/thread.jspa?messageID=683536

Brian Ecker
  • 2,017
  • 1
  • 21
  • 30
0

AWS Elasticsearch only works on port 80 using Rest API. The default transport API from elasticsearch does not work with Hosted AWS elasticsearch.

I had to customize the Jest client to add additional hooks to sign every request.

You may want to refer to this post on Stack Overflow for more details on how to sign the HTTP request.

Community
  • 1
  • 1
Yeshodhan Kulkarni
  • 2,844
  • 1
  • 16
  • 19
0

Use HOST: ip address of hosted AWS elasticsearch or path of hosted AWS es and PORT : 80

Ex:

10.100.1.1 or xyz.abc.com

Aniruddh Rathore
  • 199
  • 1
  • 5