ElasticSearch entered "read only" mode, node cannot be altered

Question

Something happened during the night to my ES cluster (composed of 5 data nodes, 3 master nodes).

I have no idea what happened but all the indices and data were deleted and the cluster entered a "read only" mode, possibly hacked?

When trying to get Kibana running, I get the following:

Tried restarting Kibana - it restarted, nothing changed. Tried restarting Elastic - it restarted (all nodes), nothing changed.

I then had a look at the cluster settings and this is what I got:

{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "all"
        }
      },
      "blocks": {
        "read_only": "true"
      }
    }
  },
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "all"
        }
      }
    }
  }
}

I tried undoing the read only as follows:

PUT _cluster/settings
{
  "persistent": {
    "blocks.read_only": false
  }
}

No luck as you can see:

{
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
      }
    ],
    "type": "cluster_block_exception",
    "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
  },
  "status": 403
}

Any ideas?

UPDATE: Problem solved by Andrei Stefan, now for the more important part - why? What happened and why? I've lost all data and my cluster entered a read-only mode.

Can you check if you have a `blocks.read_only: true` setting in your `elasticsearch.yml` file? — Val, Aug 03 '16 at 06:01

score 17 · Accepted Answer · answered Aug 03 '16 at 06:42

17

The correct command is:

PUT /_cluster/settings
{
  "persistent" : {
    "cluster.blocks.read_only" : false
  }
}

answered Aug 03 '16 at 06:42

Andrei Stefan

51,654
6
98
89

`curl -u elastic:changeme -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{"persistent":{"cluster.blocks.read_only":false}}'` unfortunately didn't work for me :( although it does change the read only property: `{"acknowledged":true,"persistent":{"cluster":{"blocks":{"read_only":"false"}}},"transient":{}}` – Rambatino Dec 10 '17 at 16:33

score 11 · Answer 2 · answered Sep 24 '18 at 15:40

11

It turns out ES has some thresholds for available disk space, and when the "flood" one is hit, it puts the indeces into read only mode.

In order to set it back (tested with ES6), you will need to do the following:

PUT /[index_name]/_settings
{
  "index.blocks.read_only_allow_delete": null
}

More information can be found on the following page of the documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html

answered Sep 24 '18 at 15:40

dg6

470
5
9

this is exactly what happened in our case! thank you for sharing – asgs Mar 13 '19 at 18:11
2

Appreciated, for resetting all indices at once could be issued something like: `curl -X PUT "localhost:9200/*/_settings" -H 'Content-Type: application/json' -d'{"index.blocks.read_only_allow_delete": null}'` – dess Jul 01 '19 at 17:17

ElasticSearch entered "read only" mode, node cannot be altered

2 Answers2

Linked