Observed Forbitten exception often in kibana dashboard

Question

I'm observing the below error message often in kibana. If I restarted the images and it looks fine for while but I'm loosing all my data as I explained in this below link.

The data is getting lost whenever I restart the docker/elk image

Error Message:

Erased the hostname in the snapshot.

I just searched and found that the below script will resolved that issue:

Script:

  PUT kindle_s_builds/_settings
     {
      "index": 
       {
        "blocks": 
        {
          "read_only_allow_delete": "false"
        }
       }
     }

Output:

  {
     "acknowledged" : true
  }

Kibana version : 6.6.0 and using docker/elk image to host the kibana dashboard.

But nothing helps. Can anyone let me know how can I resolve this issue? and what is the root cause and how to overcome this?

score 0 · Accepted Answer · answered Oct 12 '19 at 14:28

Looks like your ES cluster has gone into read-only mode due to very less disk space.

Can you check if your ES cluster is having sufficient disk space? You can run this curl command

curl -u username:password -XGET "http://ES_HOST:9200/_cat/allocation?v&pretty"

to know about the free disk space in your cluster.

Also, have a look at Elasticsearch error: cluster_block_exception [FORBIDDEN/12/index read-only / allow delete (api)], flood stage disk watermark exceeded

score 0 · Answer 2 · edited Oct 12 '19 at 17:02

There are two angles to your problem:

The indexes are becoming read only and allow delete because you ran out of disk and as a way to warn you elastic did that. You already found the solution which is to set read_only_allow_delete as false on affected indexes.
You are losing your data because whenever the container is destroyed and created again the history is lost. You need to mount a physical disk folder inside your container. You can use the -v option in the docker run command as below:

docker run -p 5601:5601 -p 9200:9200  -p 5044:5044 -v /Users/mk/data:/var/lib/elasticsearch --name elk sebp/elk

This will serve as a persistent storage for your data and every time the container is destroyed, the data will be safe on disk.

Observed Forbitten exception often in kibana dashboard

2 Answers2