79

I use Elasticsearch 1.4.4 in my development machine (a single notebook). Everything is set as default because I never changed any settings.

When I start it, I usually get the following message: 

[2015-10-27 09:38:31,588][INFO ][node                     ] [Milan] version[1.4.4], pid[33932], build[c88f77f/2015-02-19T13:05:36Z]
[2015-10-27 09:38:31,588][INFO ][node                     ] [Milan] initializing ...
[2015-10-27 09:38:31,592][INFO ][plugins                  ] [Milan] loaded [], sites []
[2015-10-27 09:38:34,665][INFO ][node                     ] [Milan] initialized
[2015-10-27 09:38:34,665][INFO ][node                     ] [Milan] starting ...
[2015-10-27 09:38:34,849][INFO ][transport                ] [Milan] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.81.1.108:9300]}
[2015-10-27 09:38:35,022][INFO ][discovery                ] [Milan] elasticsearch/DZqnmWIZRpapZY_TPkkMBw
[2015-10-27 09:38:38,787][INFO ][cluster.service          ] [Milan] new_master [Milan][DZqnmWIZRpapZY_TPkkMBw][THINKANDACT1301][inet[/10.81.1.108:9300]], reason: zen-disco-join (elected_as_master)
[2015-10-27 09:38:38,908][INFO ][http                     ] [Milan] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.81.1.108:9200]}
[2015-10-27 09:38:38,908][INFO ][node                     ] [Milan] started
[2015-10-27 09:38:39,220][INFO ][gateway                  ] [Milan] recovered [4] indices into cluster_state
[2015-10-27 09:39:08,801][INFO ][cluster.routing.allocation.decider] [Milan] low disk watermark [15%] exceeded on [DZqnmWIZRpapZY_TPkkMBw][Milan] free: 58.6gb[12.6%], replicas will not be assigned to this node
[2015-10-27 09:39:38,798][INFO ][cluster.routing.allocation.decider] [Milan] low disk watermark [15%] exceeded on [DZqnmWIZRpapZY_TPkkMBw][Milan] free: 58.6gb[12.6%], replicas will not be assigned to this node
[2015-10-27 09:40:08,801][INFO ][cluster.routing.allocation.decider] [Milan] low disk watermark [15%] exceeded on [DZqnmWIZRpapZY_TPkkMBw][Milan] free: 58.6gb[12.6%], replicas will not be assigned to this node
....

I see a lot of these "low disk watermark ... exceeded on..." messages. What went wrong in my case? How to fix it? Thanks!

UPDATE

Before this post, I searched SO for related posts. I found one related to "high watermark..." and in that case, the disk space is low. In my case, I checked and there is still 56GB left on my disk.

UPDATE

According to the input from Andrei Stefan, I need to change settings. Should I do it the following way:

curl -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.disk.threshold_enabled" : false
    }
}'

Or is there any settings file I can edit to set it?

slm
  • 15,396
  • 12
  • 109
  • 124
curious1
  • 14,155
  • 37
  • 130
  • 231
  • 1
    Read about this here: https://www.elastic.co/guide/en/elasticsearch/reference/current/disk.html, the exception message is self explanatory, why it's like that it's not so self-explanatory. That section of the documentation should clear things. – Andrei Stefan Oct 27 '15 at 14:00
  • 2
    @curious1 ES takes some decisions based on disk space available. How ES decides what "large" or "small" means in terms of diskspace is defined by those high and low watermarks. If you are not happy with those percentages, in some situations it's advisable to use specific values. For example, if you have a disk that 4TB large, 85% low watermark means 600GB free space. 600GB is enough for some large indices. In this case you can specify a low watermark as 30GB and high watermark as 100GB or similar. On my computer, I am always on low watermark warning ;-), this doesn't mean I'm not using ES. – Andrei Stefan Oct 27 '15 at 14:26
  • 2
    @curious1 That command you have there is working on a `transient` setting, meaning when you restart the cluster the setting is gone. Make it `persistent` to survive restarts or put it in `elasticsearch.yml` files like this: `cluster.routing.allocation.disk.threshold_enabled: false`. – Andrei Stefan Oct 27 '15 at 23:29

5 Answers5

100

If you like me have a lot of disk you can tune the watermark setting and use byte values instead of percentages:

NB! Use either percentage values or byte values.

You cannot mix the usage of percentage/ratio values and byte values within the watermark settings. Either all values are set to percentage/ratio values, or all are set to byte values.

Setting: cluster.routing.allocation.disk.watermark.low

Controls the low watermark for disk usage. It defaults to 85%, meaning that Elasticsearch will not allocate shards to nodes that have more than 85% disk used. It can alternatively be set to a ratio value, e.g., 0.85. It can also be set to an absolute byte value (like 500mb) to prevent Elasticsearch from allocating shards if less than the specified amount of space is available. This setting has no effect on the primary shards of newly-created indices but will prevent their replicas from being allocated.

Setting: cluster.routing.allocation.disk.watermark.high

Controls the high watermark. It defaults to 90%, meaning that Elasticsearch will attempt to relocate shards away from a node whose disk usage is above 90%. It can alternatively be set to a ratio value, e.g., 0.9. It can also be set to an absolute byte value (similarly to the low watermark) to relocate shards away from a node if it has less than the specified amount of free space. This setting affects the allocation of all shards, whether previously allocated or not.

Setting:: cluster.routing.allocation.disk.watermark.flood_stage

Controls the flood stage watermark, which defaults to 95%. Elasticsearch enforces a read-only index block (index.blocks.read_only_allow_delete) on every index that has one or more shards allocated on the node, and that has at least one disk exceeding the flood stage. This setting is a last resort to prevent nodes from running out of disk space. The index block is automatically released when the disk utilization falls below the high watermark. Similarly to the low and high watermark values, it can alternatively be set to a ratio value, e.g., 0.95, or an absolute byte value.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#disk-based-shard-allocation

Please note:

Percentage values refer to used disk space, while byte values refer to free disk space. This can be confusing, since it flips the meaning of high and low. For example, it makes sense to set the low watermark to 10gb and the high watermark to 5gb, but not the other way around.

On my 5TB disk I've set:

# /etc/elasticsearch/elasticsearch.yml
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.flood_stage: 5gb
cluster.routing.allocation.disk.watermark.low: 30gb
cluster.routing.allocation.disk.watermark.high: 20gb
sastorsl
  • 2,015
  • 1
  • 16
  • 17
  • 3
    Please note that `cluster.routing.allocation.disk.threshold_enabled: True` should be `cluster.routing.allocation.disk.threshold_enabled: true` (without capital T) – seinecle Apr 25 '17 at 21:35
  • Strictly speaking it is not an _error_ - meaning both True and true works, even if the java standard is to de-capitalize. At the time of writing the problem at hand was more important than adhering to a coding style - i would assume. It's been awhile... – sastorsl May 04 '17 at 18:13
  • I think in my case it did not work, that's why I was commenting. – seinecle May 10 '17 at 11:52
  • 13
    Note that if you want to set percentages they should be ratios. E.g. `cluster.routing.allocation.disk.watermark.low: 0.95` – arno_v May 24 '17 at 11:29
  • There is another relevant setting cluster.routing.allocation.disk.watermark.flood_stage which is above the high watermark. i.e. in the example above it would have to be <20gb. – Bruce Adams Jan 15 '19 at 13:39
  • If Elasticsearch reports that high watermark still at 90% even after you set these settings, then try to set these settings in elasticsearch.yml of every master node. It worked for me, though I don't sure if it was the solution – Amaimersion Mar 31 '23 at 18:00
44

I know it is old post, but my comment can make someone happy. In order to specify watermark in bytes values (gb or mb) you have to add cluster.routing.allocation.disk.watermark.flood_stage to your elasticsearch settings file - elasticsearch.yml. Complete example: 

  cluster.routing.allocation.disk.threshold_enabled: true 
  cluster.routing.allocation.disk.watermark.flood_stage: 200mb
  cluster.routing.allocation.disk.watermark.low: 500mb 
  cluster.routing.allocation.disk.watermark.high: 300mb

Note: without specifying cluster.routing.allocation.disk.watermark.flood_stage it will not work with bytes value (gb or mb)

Milad Hamid Elhaei Sahar
  • 543
  • 8
  • 12
24

In my case - I had just to turn off threshold:

run ElasticSearch:

elasticsearch

On other tab run:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'


curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

macOS Catalina, ElasticSearch installed via Brew.

iyanglow
  • 351
  • 2
  • 2
14

I added the following lines to my elasticsearch.yaml file (elastic_search_folder\config):

cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 93%
cluster.routing.allocation.disk.watermark.high: 95%

For some reason, specifying watermark in gb like in the other answer didn't work for me. Also, make sure your watermark.high is less or equal to the flood watermark (it's usually set to 95%).

This worked for me in version 6.1.1.

TylerH
  • 20,799
  • 66
  • 75
  • 101
Ivan Yurchenko
  • 3,762
  • 1
  • 21
  • 35
  • 1
    You have to set all 4 settings you cannot mix percentages + mb/gb. You're missing the `cluster.routing.allocation.disk.watermark.flood_stage` setting which was likely blocking you from using the mb/gb settings. – slm May 13 '19 at 16:36
0

I want to add some background on why Elasticsearch has these various thresholds and what can happen if these threshold crossed.

Background

Elasticsearch considers the available disk space before deciding whether to allocate new shards, relocate shards away or put all indices on read mode based on a different threshold of this error. The reason is Elasticsearch indices consists of different shards which are persisted on data nodes.

Solution

My detailed blog on relevant settings and temporary and permanent fix can be found here.

In short, permanent fixes are:

  1. Delete unused indices.
  2. Merge segments using the answers from Optimize API for reducing the segments and eliminating ES deleted docs not working.
  3. Attach external disk or increase the disk used by the problematic data node.
TylerH
  • 20,799
  • 66
  • 75
  • 101
Amit
  • 30,756
  • 6
  • 57
  • 88