high disk watermark [90%] exceeded [...] shards will be relocated away from this node

Question

I'm trying to run multi node Elasticsearch from docker containers in my local machine. Below is my docker compose file:

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.13.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - C:/Docker/Elasticsearch/data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.13.0
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - C:/Docker/Elasticsearch/data02:/usr/share/elasticsearch/data
    networks:
      - esnet
  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.13.0
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - C:/Docker/Elasticsearch/data03:/usr/share/elasticsearch/data
    networks:
      - esnet
  kibana:
    image: docker.elastic.co/kibana/kibana:7.13.0
    container_name: kibana
    environment:
      - "ELASTICSEARCH_HOSTS=http://es01:9200"
    ports:
      - '5601:5601'
    networks:
      - esnet

networks:
  esnet:
    driver: bridge

When I'm running the docker compose command then I'm getting below error:

es01      | {"type": "server", "timestamp": "2021-05-29T12:08:16,409Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "high disk watermark [90%] exceeded on [6UfoOc1-QaCrrRlhLngSkA][es02][/usr/share/elasticsearch/data/nodes/0] free: 17.3gb[7.3%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high 
disk watermark when these relocations are complete", "cluster.uuid": "dpml3lE2Q0i7NRxFaQcGkQ", "node.id": "F_delTaoRfCASAlmu_Yd-Q"  }

Below is the screenshot of my docker resources:

The below the docker server message from http://127.0.0.1:9200/

{
  "name" : "es01",
  "cluster_name" : "es-docker-cluster",
  "cluster_uuid" : "dpml3lE2Q0i7NRxFaQcGkQ",
  "version" : {
    "number" : "7.13.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "5ca8591c6fcdb1260ce95b08a8e023559635c6f3",
    "build_date" : "2021-05-19T22:22:26.081971330Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

But, my Kibana is not working. :(

In your Docker compose, you mount the data location to your local disk (`C:/Docker/Elasticsearch/...`), so I suspect you are running out of disk space there. Based on the error message (`free: 17.3gb[7.3%]`) looks like you are using a disk which roughly of the size **~236 GB** and you only have **~17.3 GB** free space left. Can you check? — zsltg, May 30 '21 at 07:26
@zsltg, yes you are correct, and after I had to clean up some of the disk space and then encountered another issue related to virtual memory. I got that fixed by using https://stackoverflow.com/a/66547784/596495 answer. — Roul, May 31 '21 at 02:57

score 3 · Accepted Answer · answered May 31 '21 at 03:06

In the comment section, I have already mentioned what was my issue and just for future reference I'm explaining it here:

high disk watermark [90%] exceeded […] shards will be relocated away from this node: This error will come whenever you are encountering low disk space available in your system. So, just to overcome this issue I had to clean up some data to make more disk space and my current free space is 40GB which is working fine.
After fixing the Disk space I countered another issue that was "Max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]".
So here we need to increase the virtual memory of the Docker. In windows first, we need to go to the Docker terminal before executing the increased virtual memory command.

If your Docker using wsl subsystem then

open power shell
Run: wsl -d docker-desktop, this will take to Docker terminal.
Run: sysctl -w vm.max_map_count=262144

Restart your Docker and you are all set.

Note: The increase the virtual memory is already answered here: https://stackoverflow.com/a/66547784/596495

Note: for me on Windows I had to increase "Disk image size" under Docker Desktop > Settings > Resources > Advanced as the 64GB default was nearly full. Although my host had plenty of free disk space _and_ I had a Docker volume mounted to `/usr/share/elasticsearch/data` the problem is that Docker Desktop on Windows (at least when _not_ using WSL 2) is basically running your containers in a linux Hyper-V VM and that is what was running out of disk space. — sparrowt, Jan 23 '23 at 11:51

score 0 · Answer 2 · answered Jan 06 '23 at 03:49

This error message is also displayed when you do not mount a volume to /usr/share/elasticsearch/data. ElasticSearch is trying to add data to the default filesystem of the container and fails.

The OP has mounted a volume, but search engines bring this post at the top so I'm posting it here. You can solve it with adding a volume using the -v switch for the docker command, or adding a volume in docker-compose.yml:

    volumes:
      - ${HOST_DATA_DIR}/elasticsearch:/usr/share/elasticsearch/data

high disk watermark [90%] exceeded [...] shards will be relocated away from this node

2 Answers2