How to add a new node to my Elasticsearch cluster

Question

My cluster has a yellow health as it has only one single node, so the replicas remain unasigned simply because no other node is available to contain them.

So I want to create/add another node so Elasticsearch can begin allocating replica’s to it. I've only one machine and I'm running ES as a service.

I've found tons of site with some info but none of them is giving me clearly how can I add another node to ES.

Can someone explain me which files do I've to edit and what commands do I've to launch in order to create another node in my cluster? Do I've to run two ES instance? How can I do this?

Thanks in advance.

Answer 1

TIPS TO ADD ANOTHER NODE:

1) VERSIONS:

It is a good advise to check all of your nodes for the status: http://elastic-node1:9200/

Keep in mind that in most cases: VERSION NEED TO BE THE SAME, EVEN MINOR

{
"name" : "node2",
"cluster_name" : "xxxxxxxxxxx",
"cluster_uuid" : "n-xxxxxxxxxxxxxxx",
"version" : {
  "number" : "5.2.2",
  "build_hash" : "xxxx",
  "build_date" : "20-02-24T17:26:45.835Z",
  "build_snapshot" : false,
  "lucene_version" : "6.4.1"
},
"tagline" : "You Know, for Search"
}

Keep in mind that if you see a different version number in node1, e.g.

  "number" : "5.2.1"

you have to update your node in that case to version 5.2.2 (same as node1).

2) NODES AND REPLICA:

What is the usecase of the node? For 3 nodes I would do this:

curl -XPUT 'localhost:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
{
  "transient": {
    "discovery.zen.minimum_master_nodes": 3
  }
}
'

Even better is to change settings in Elasticsearch's configuration file:

/etc/elasticsearch/elasticsearch.yml 

# need to be changed on each node (has to be unique for each node):
node.name: node1

# need to be the same in all nodes:
cluster.name: my_cluster
discovery.zen.ping.unicast.hosts: ["IP_ADDRESS_OR_HOSTNAME1", "IP_ADDRESS_OR_HOSTNAME2", "IP_ADDRESS_OR_HOSTNAME3"]

And if you have 3 nodes, do you want two replicas and one primary?

curl -XPUT 'localhost:9200/_settings?pretty' -H 'Content-Type: application/json' -d'
{
    "index" : {
        "number_of_replicas" : 2
    }
}'

3) MAKE SURE THAT NODES ARE ENABLED

There is a way to kick a node:

curl -XPUT localhost:9200/_cluster/settings -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._ip" : "NODE_TO_REMOVE_IP_ADDRESS_OR_HOSTNAME"
   }
}';echo

So if you did that, and now you want to add the node back: https://www.elastic.co/guide/en/elasticsearch/guide/current/_rolling_restarts.html

you can do that with following request (please read carefully mentioned link above):

curl -XPUT localhost:9200/_cluster/settings -d '{
  "transient" :{
        "cluster.routing.allocation.enable" : "all"
   }
}';echo

4) NEVER FORGET, NETWORKING:

Firewall, network... Can you reach the new node at port 9200? Can you see it on your web browser?

Can you

curl http://your-elasticsearch-hostname:9200/

?

TIPS TO REMOVE NODE FROM CLUSTER:

1) REMOVE WITH API

curl -XPUT 'http://localhost:9200/_cluster/settings?pretty' -d '
{
  "transient" : {
    "cluster.routing.allocation.exclude._name" : "node3"
  }
}'

2) CHECK YOUR CONFIG FILE

Check config file under: /etc/elasticsearch/elasticsearch.yml

and leave only the nodes you want to keep:

discovery.zen.ping.unicast.hosts:["IP_ADDRESS_OR_HOSTNAME1", "IP_ADDRESS_OR_HOSTNAME2"]

* CHECK YOUR STATUS *

Check http://elk-pipeline:9200/_cat/shards What is your status? You may see: INITIALIZING That probably means that data is transferred. So if your data is large, (and not on SSD), wait.

DON'T FORGET

You can see if your data is currently moving by typing:

[watch] du /var/lib/elasticsearch/

That is all for now. I will try to add more information here from time to time.

Please feel free to change/add.

Answer 2

Complete steps on Windows Box are:

1. unzip elastic, say, to C:\ELK\elastic\ run command bin\service install elastic-search-x64-node01 which will create service named elastic-search-x64-node01

2. edit elasticsearch.yml config file:

   cluster.name: Animals
   node.name: Snake
   node.master: true
   node.data: true
   path.data: C:\ELK\storage\data
   path.logs: C:\ELK\storage\logs
   http.port: 9200
   discovery.zen.ping.multicast.enabled: false
   discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
   

3. run service manager elastic-search-x64-node01 and set-up your services rules and start the service

4. unzip elastic, say, to C:\ELK\elastic2\ run command bin\service install elastic-search-x64-node02 which will create service named elastic-search-x64-node02

5. edit elasticsearch.yml config file:

   cluster.name: Animals
   node.name: Baboon
   node.master: false
   node.data: true
   path.data: C:\ELK\storage\data
   path.logs: C:\ELK\storage\logs
   http.port: 9201
   discovery.zen.ping.multicast.enabled: false
   discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
   

6. run service manager elastic-search-x64-node02 and set-up your services rules and start the service

At this point you have 2 separate nodes (they store data in the same folder, but I'm too laze to edit path.data for second node) as 2 separate Windows Services and GET http://localhost:9200/_cluster/health shows something like:

{
  "cluster_name": "Animals",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 2,
  "number_of_data_nodes": 2,
  "active_primary_shards": 4,
  "active_shards": 8,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

Answer 3

First, you can remove the replicas to go back to a green state, you can do this even after you have created the index and added documents to it.

Here's how you set the replica count to 0:

curl -XPUT 'localhost:9200/my_index/_settings' -d '
{
    "index" : {
        "number_of_replicas" : 0
    }
}'

If you would like to simply add another node to your cluster, you will need to edit the elasticsearch.yml, preferably on both of your nodes with these changes:

cluster.name: my-cluster
node.name: node01
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["192.168.0.5"]

Set the unicast hosts on each node to reference the other, set the cluster name to be the same on both nodes and give each node a unique name, then restart both ES instances and your cluster should be online.

Answer 4

I'm not clear what your question is here. There are two possibilities that you are going for.

1) You have two nodes, and you want them to be able to see each other. There's a lot of documentation for this around all over the place. If this is the case, I would suggest you test to make sure you each host can talk to the other, ie: ssh into host A and try to curl host B:9200, and vice versa. You'll also want to make sure you configurations are set correctly.

2) You want to set up a 'cluster' on your local machine. In this case, you'll need two separate config files, and you'll need to start up elasticsearch using a specific configuration file for your second 'node' (as well as modify the second configuration to use different ports, etc). You might want to check out this link: http://www.concept47.com/austin_web_developer_blog/elasticsearch/how-to-run-multiple-elasticsearch-nodes-on-one-machine/

But yes, ultimately, you need to run to elasticsearch processes, whether they are both on the same machine, or two different machines is up to you

Answer 5

If you are running Elasticsearch n local just go to another terminal and run elasticsearch again. This way you will have two instance.

You will see following message on the newly run instance:

[Black Widow] detected_master [N'Garai]

On Already running instance(master node)

[N'Garai] added {[Black Widow]

Assuming [N'Garai] and {[Black Widow] to be random node names.