All shards failed

Question

I was working on elastic search and it was working perfectly. Today I just restarted my remote server (Ubuntu). Now I am searching in my indexes, it is giving me this error.

{"error":"SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed]","status":503}

I also checked the health. The status is red. Can anyone tell me what's the issue.

score 56 · Answer 1 · answered Jan 16 '14 at 12:07

56

It is possible on your restart some shards were not recovered, causing the cluster to stay red.
If you hit:
http://<yourhost>:9200/_cluster/health/?level=shards you can look for red shards.

I have had issues on restart where shards end up in a non recoverable state. My solution was to simply delete that index completely. That is not an ideal solution for everyone.

It is also nice to visualize issues like this with a plugin like:
Elasticsearch Head

answered Jan 16 '14 at 12:07

mconlin

8,169
5
31
37

1

Hi @mconlin, how do you figure which index to be deleted in this case? – Nikhil Mulley Apr 28 '14 at 17:20
using head you will see greyed out unrecovered shards on the last row. – mconlin Apr 29 '14 at 20:56
If you are on docker, try to force recreate elasticsearch and kibana – paaacman Jan 19 '19 at 23:18
Note that a large number of shards will take a long time to initialize; try reducing the number, see: https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster – slhck Feb 12 '19 at 08:43

Paulo Victor · Answer 2 · 2020-04-01T17:41:38.797

43

If you're running a single node cluster for some reason, you might simply need to do avoid replicas, like this:

curl -XPUT -H 'Content-Type: application/json' 'localhost:9200/_settings' -d '
{
    "index" : {
        "number_of_replicas" : 0
    }
}'

Doing this you'll force to use es without replicas

edited Apr 01 '20 at 17:41

answered Mar 11 '19 at 22:05

Paulo Victor

3,814
2
26
29

what will this do?? @paulo , pls explain – AATHITH RAJENDRAN Nov 07 '19 at 12:19
1

Tells ES that all your indices are on a single machine: no replicas. – Marc Feb 10 '20 at 19:50
I had one machine and one elasticsearch node installed on it without replicas and this command worked for me. – Vahid F Dec 07 '20 at 10:14

score 5 · Answer 3 · edited Jan 20 '23 at 19:16

first thing first, all shards failed exception is not as dramatic as it sounds, it means shards were failed while serving a request(query or index), and there could be multiple reasons for it like

Shards are actually in non-recoverable state, if your cluster and index state are in Yellow and RED, then it is one of the reasons.
Due to some shard recovery happening in background, shards didn't respond.
Due to bad syntax of your query, ES responds in all shards failed.

In order to fix the issue, you need to filter it in one of the above category and based on that appropriate fix is required.

The one mentioned in the question, is clearly in the first bucket as cluster health is RED, means one or more primary shards are missing, and my this SO answer will help you fix RED cluster issue, which will fix the all shards exception in this case.

score 3 · Answer 4 · edited Dec 31 '20 at 14:59

3

If you encounter this apparent index corruption in a running system, you can work around it by deleting all files called segments.gen. It is advisory only, and Lucene can recover correctly without it.

From ElasticSearch Blog

edited Dec 31 '20 at 14:59

LhasaDad

1,786
1
12
19

answered Jul 23 '14 at 11:13

chemark

1,181
1
13
19

2

The current link is redirecting to the main elastic.co page. It no longer shows the blog entry. Edit submitted. – LhasaDad Dec 31 '20 at 03:39

score 0 · Answer 5 · edited Aug 23 '23 at 12:24

0

For Elasticsearch > 5.0 it's possible to get some more information from this endpoint:

http://localhost:9200/_cluster/allocation/explain?pretty

I just ran into a case where I hit the virtual disk limit configured in Docker Desktop and adding an additional, unrelated container caused ES to fail.

edited Aug 23 '23 at 12:24

Roopendra

7,674
16
65
92

answered Apr 29 '23 at 20:24

Alex

2,398
1
16
30

score 0 · Answer 6 · answered Jul 03 '23 at 23:39

0

If you are upgrading the Elasticsearch and have multiple versions you can face this issue. Continue to upgrade ALL nodes. And run the daemon reload.

sudo systemctl daemon-reload

answered Jul 03 '23 at 23:39

Musab Dogan

1,811
1
6
8

All shards failed

6 Answers6

Linked