1

I have my Elasticsearch cluster in a VPC, I'd like to access this EC cluster from my local Macbook. I have set up a bastion host that uses the same VPC and the same security group, and I was able to ssh into this bastion host from my Macbook.

But somehow, my code just cannot connect to my ES cluster through this bastion host, here's my command to run port forwarding:

ssh -i ~/Downloads/keypairs/20210402-02.pem ubuntu@ec2-123-456.us-west-2.compute.amazonaws.com -N -L 9200:vpc-es-domain-20210331-abc123def.us-west-2.es.amazonaws.com:443

Here's my timeout exception when accessing the ES cluster in the VPC:

java.net.ConnectException: Timeout connecting to [vpc-es-domain-20210331-abc123def.us-west-2.es.amazonaws.com/10.0.47.182:443]
    at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:823) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514) ~[elasticsearch-rest-high-level-client-7.6.1.jar:7.6.1]
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484) ~[elasticsearch-rest-high-level-client-7.6.1.jar:7.6.1]
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454) ~[elasticsearch-rest-high-level-client-7.6.1.jar:7.6.1]
    at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497) ~[elasticsearch-rest-high-level-client-7.6.1.jar:7.6.1]

Here are the rules of my SG:

Inbound:

All TCP TCP 0 - 65535   0.0.0.0/0
All traffic All All sg-abc123 / default
SSH TCP 22  0.0.0.0/0

Outbound:

All traffic All All 0.0.0.0/0

When I've ssh'ed into my bastion host, and run curl vpc-es-domain-20210331-abc123def.us-west-2.es.amazonaws.com, I got this response:

{
  "name" : "abc123",
  "cluster_name" : "abc123097:es-domain-beta-20210331",
  "cluster_uuid" : "abc123def",
  "version" : {
    "number" : "7.8.0",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2021-01-15T06:15:47.944536Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

I'm suspecting my command to do port forwarding is not correct? But after research, this looks the most legit option to me. Any insight would be greatly appreciated!

Fisher Coder
  • 3,278
  • 12
  • 49
  • 84
  • Maybe you didn't authorize your bastion security group in the ElasticDomain security group – Bertrand Martel Apr 04 '21 at 17:48
  • thanks for the comment, but I actually launched my bastion host using the same SG as my ES, updated my OP as well. Any other insights? – Fisher Coder Apr 04 '21 at 18:05
  • did you authorize access from this same security group for the port 443 ? – Bertrand Martel Apr 04 '21 at 18:06
  • eg add a rule with port 443 and as source: [this security group ID] – Bertrand Martel Apr 04 '21 at 18:09
  • ok, I've added this `All TCP TCP 0 - 65535 0.0.0.0/0` as a new inbound rule to my SG associated with this bastion host. Still no luck. – Fisher Coder Apr 04 '21 at 18:17
  • do you have access to the ES cluster when you ssh on the bastion host? On the bastion host: `curl https://vpc-es-domain-20210331-abc123def.us-west-2.es.amazonaws.com` is it working ? If not and you're sure the SG rules are correct, it may be a problem with the ES cluster – Bertrand Martel Apr 04 '21 at 18:31
  • interesting, that one gave me an error, I've updated the OP to reflect this. I'm looking into it now – Fisher Coder Apr 04 '21 at 18:37
  • I've just tested with my ES cluster, this works well `ssh -NL 9200::[ES URL]:443 ec2-user@[EC2 URL] -i key.pem` which looks like exactly like your ssh command. I can access `https://localhost:9200` in my browser with it – Bertrand Martel Apr 04 '21 at 18:55
  • I think the problem is with your ES cluster maybe check the logs if you've [configured logging](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createdomain-configure-slow-logs.html) – Bertrand Martel Apr 04 '21 at 18:58
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/230736/discussion-between-fisher-coder-and-bertrand-martel). – Fisher Coder Apr 04 '21 at 19:44

1 Answers1

1

The code running on your local computer is trying to connect directly to the Elasticsearch server without going through the SSH tunnel. The SSH command is opening a tunnel from your local port 9200 to the remote server. The local software trying to connect to Elasticsearch should be connecting to localhost:9200 not vpc-es-domain-20210331-abc123def.us-west-2.es.amazonaws.com/10.0.47.182:443.

The endpoint vpc-es-domain-20210331-abc123def.us-west-2.es.amazonaws.com/10.0.47.182:443 doesn't look valid anyway. It has a hostname and an IP address in there.

You mentioned in the comments:

"I actually launched my bastion host using the same SG as my ES"

However just placing two resources in the same security group does nothing unless that security group also has a rule specifically allowing traffic between the resources within it. Security groups do not have this rule by default, except for the default security group in the default VPC that is created automatically when you first create your AWS account.

So please make sure that the security group has a rule that will allow the bastion host to connect to the Elasticsearch server over port 443.

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • Thanks Mark. I've just added an inbound rule that does so, I've updated in the OP, please let me know if that looks correct to you? – Fisher Coder Apr 04 '21 at 18:25
  • So, how can I make my local code to connect to the ES server via the SSH tunnel instead of directly? – Fisher Coder Apr 04 '21 at 18:25
  • @FisherCoder per all the comments and chat, you're getting "Address already in use" because you already ran that command once. You already have the ssh tunnel setup. You are also getting a response from Elasticsearch when you curl from your bastion host, so the security group is configured correctly. All you need to do is change the code running on your local computer to connect to `https://localhost:9200` as I stated in my answer. – Mark B Apr 04 '21 at 20:05
  • could you list out all the steps that I needed to do to access the ES cluster from my Mac? So far, I can access my localhost ES service through my code through HTTP instead of HTTPS. But what I'd like to achieve is to access the ES in the VPC as I stated in my OP. Thanks – Fisher Coder Apr 04 '21 at 20:09
  • 1
    Step 1. shutdown your local ES service. Step 2. Start the ssh tunnel if it isn't already running. Step 3. Access `https://localhost:9200` which is your local end of the SSH tunnel, which will send your requests to the AWS ES service. – Mark B Apr 04 '21 at 20:21
  • Got it, these three steps helped clarify things a lot! Thanks! Now, when I'm running it, I got this error: `Caused by: javax.net.ssl.SSLPeerUnverifiedException: Host name 'localhost' does not match the certificate subject provided by the peer (CN=*.us-west-2.es.amazonaws.com)` , a quick search led me to this: https://stackoverflow.com/questions/37879751/spring-rest-template-host-name-localhost-does-not-match-the-certificate-subj , I guess I should follow? – Fisher Coder Apr 04 '21 at 20:30
  • Yes you'll probably need to use one of those options that disables SSL certification validation, using `TrustSelfSignedStrategy()`. Alternatively you could add a record to your Mac's `/etc/hosts` file. – Mark B Apr 04 '21 at 20:33
  • Great, could you elaborate a bit more what record to add in my `/etc/hosts` file? Many thanks! – Fisher Coder Apr 04 '21 at 20:35
  • 1
    Nvm, I've figured out this error as well by setting this: `httpClientBuilder.setSSLHostnameVerifier((s, sslSession) -> true);`, thanks a lot to you both! – Fisher Coder Apr 04 '21 at 21:07