13

I have set up a elasticsearch server using AWS elasticsearch service (Not EC2). It gave me an endpoint https://xxx-xxxxxxxx.us-west-2.es.amazonaws.com/ and if I click this endpoint(Note that there is no port specified) I can get the expected

{
  status: 200,
  name: "Mastermind",
  cluster_name: "xxxx",
  version: {
    number: "1.5.2",
    build_hash: "yyyyyy",
    build_timestamp: "2015-04-27T09:21:06Z",
    build_snapshot: false,
    lucene_version: "4.10.4"
  },
  tagline: "You Know, for Search"
}

The question is how do I get this through the elasticsearch java client without a port number? The sample code I get is

Client client = TransportClient.builder().build()
    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("host1"), 9300));

If I use this code and just replace "host1" with my endpoint, I'll get "NoNodeAvailableException"

ps: The java client version I'm using is 2.0.0.

Edit I finally decided to go with Jest, a 3rd party REST client. But what Brooks answered below is also very helpful - AWS do use port 80 for http and 443 for https. The blocker for me was the firewall I guess.

Edit2 The AWS ES service documentation explicitly says:
The service supports HTTP on port 80, but does not support TCP transport.

Edmond
  • 614
  • 2
  • 11
  • 26
  • 1
    If my answer satisfied your question, I would appreciate it if you marked it as correct. – Brooks Nov 07 '15 at 16:05
  • @Brooks, I still could not connect to AWS elastic service using the transport client and finally decided to go with Jest. But your answer still helped me understanding what the issue is. I've edited my question accepted your answer. If you find out a complete solution, please edit your answer so that other people can benefit. – Edmond Nov 07 '15 at 23:52
  • Hi Edmond, you still couldn't connect? Your access policy is in place? – Brooks Nov 08 '15 at 00:33
  • @Brooks, Yes it's in place. Actually I set it to open to public just for development and testing purpose. still didn't work. But so far JEST has been working well. – Edmond Nov 08 '15 at 06:33
  • And I assume you changed the port to 80? I'll give it a try with my ES instance tomorrow and see what happens. Never used JEST before, but sounds neat. Plus it looks like it offers SSL? Still need to have a way to decrypt it I assume. – Brooks Nov 08 '15 at 07:05
  • @Brooks, yes I did try 80 too. I heard that SaaS providers won't work with native API. – Edmond Nov 10 '15 at 01:20
  • JEST works smoothly since it's simply a nice wrapper of the standard REST API of ElasticSearch. Just trying to add a point in case that it can save others days of effort battling with the relatively new and "documentation-less" AWS ElasticSearch "Service". – CrazyGreenHand Nov 24 '15 at 12:40
  • Hello, Edmond. Sorry for late comment, but could you please clarify page in the documentation, where 80 port mentioned? – dondublon May 10 '18 at 13:38

3 Answers3

17

Believe it or not, AWS doesn't launch Elasticsearch using 9200 and 9300. It's launched via plain old port 80.

So, to demonstrate, try this...

curl -XPOST "http://xxx-xxxxxxxx.us-west-2.es.amazonaws.com:80/myIndex/myType" -d '["name":"Edmond"}'

Or

curl -XPOST "https://xxx-xxxxxxxx.us-west-2.es.amazonaws.com:443/myIndex/myType" -d '["name":"Edmond"}'

It should respond with: {"_index":"myIndex","_type":"myType","_id":"SOME_ID_#","_version":1,"created":true}

Check in Kibana and you'll see it's there.

So, then in your code, it should be:

Client client = TransportClient.builder().build()
    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("xxx-xxxxxxxx.us-west-2.es.amazonaws.com"), 80));

Unfortunately, I don't off-hand know how to transmit encrypted via SSL/HTTPS using the transport client. You could try using regular REST calls instead using JERSEY.

Finally, make sure your Elasticsearch access policy is configured properly. Something along the lines of:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:yyyyyyy:domain/myDomain/*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:yyyyyyyyy:domain/myDomain"
    }
  ]
}

NOTE: The above access policy is completely wide open and is not recommended for anything remotely close to production. Just so you know....

Brooks
  • 7,099
  • 6
  • 51
  • 82
  • yes. curl gave me Unknown SSL protocol error in connection to xxxxxxx.us-west-2.es.amazonaws.com:-9847. not sure what 9847 is. – Edmond Nov 06 '15 at 02:44
  • I think you're probably right. I can get response by hitting http://xxx.amazonaws.com:80. but I still don't know how to do that in code. BTW, InetAddress.getByName(String) cannot take "http" or "https", otherwise it will give "Unknown host" error. – Edmond Nov 06 '15 at 02:50
  • Ah, my apologies. Yeah, if you're using https, then it would be port 443. If you choose to go with http, then it would be port 80. I've updated my answer accordingly. By the way, make sure you have set the access policy for elasticsearch properly. I will update my answer for this as well. – Brooks Nov 06 '15 at 02:51
  • Right, I forgot about that. Unfortunately, I don't know how to transmit encrypted using transport client. As I mentioned in the latest edit to my answer, you would be able to do so easily using REST calls with JERSEY, but then obviously lose out on the benefits of the transport client. There are wrappers out there that support SSL for Elastic Found, but I haven't seen one for AWS. For testing purposes, I would stick with in the clear via port 80 while continuing to search for a way to encrypt. Java has to have a way, I just don't know it. – Brooks Nov 06 '15 at 03:03
3

Managed elastic search service in AWS does not provide the port for the transport protocol until now.
This question has been answered here ,

Elastic Transport client on AWS Managed ElasticSearch1

There is also a discussion in the AWS forum regarding the transport protocol. Here is the link

What is the port for the transport protocol ?

Community
  • 1
  • 1
Shahin Alam
  • 73
  • 14
1

After lot of search i found an example which used a GET request, so I made minor changes to it for allowing POST requests so that complex queries can be submitted via POST body. The implementation is available at https://github.com/dy10/aws-elasticsearch-query-java

Apart from properly configuring access to you AWS ES (i.e. dont open it to Public), make sure to use https (the above code uses http; just replace http with https in the code and it will work).

Another useful looking but partial implementation is at https://github.com/aws/aws-sdk-java/issues/861

dy10
  • 117
  • 1
  • 8
  • This is just an example of how to send HTTP POST queries to AWS ElasticSearch service. The EsQuery.java file contains the main method and you can run it directly, after replacing the 4 static String variables with real value. – dy10 May 22 '17 at 17:00
  • Your comment should be included in your answer. Comments are short-lived and can be deleted at any time. Please [edit] your answer to the missing information. – Artjom B. May 23 '17 at 20:01
  • Good to see [your access details hardcoded in the client](https://github.com/dy10/aws-elasticsearch-query-java/blob/master/src/main/java/dy/aws/es/EsQuery.java#L30), – wikier Oct 13 '17 at 22:03
  • did u read the comments 2 lines above of credentials? Those are fake credentials. It is just there so people who are less conversant can easily identify what kind of value is expected there. – dy10 Oct 15 '17 at 05:22