Elasticsearch 7 The [standard] token filter has been removed

Question

I am attempting to upgrade to Elasticsearch v7 (I'm using the ruby/rails client), and upon doing so and fixing several stuff, I run across the following error

Elasticsearch::Transport::Transport::Errors::BadRequest:
  [400] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The [standard] token filter has been removed."}],"type":"illegal_argument_exception","reason":"The [standard] token filter has been removed."},"status":400}

Upon checking the breaking changes, it is indeed mentioned that

The standard token filter has been removed because it doesn’t change anything in the stream.

I am not sure how I should reflect this in my config. If I understand correctly, this error may come from my custom phrase suggester

{
  "analysis": {
    "filter": {
      "shingle": {
        "type": "shingle",
        "min_shingle_size": 2,
        "max_shingle_size": 3
      }
    },
    "analyzer": {
      "trigram": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["standard", "shingle"]
      },
      "reverse": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["standard", "reverse"]
      }
    }
  }
}

Should I just remove the tokenizer field ? Maybe they forgot to update it but this is still what's mentionned in the [Elasticsearch documentation of the suggester][1]

If the problem is not coming from there, where should I look into ?

ES 7.3.2
elasticsearch-api-7.3.0 | elasticsearch-transport 7.3.0
elasticsearch-model 7.0.0 | elasticsearch-rails 7.0.0

I think you just need to remove `standard` from the `filter` part as described in [this](https://stackoverflow.com/a/16965481/290460) answer. — Rob, Oct 02 '19 at 13:21
@Rob am a bit noob to elasticsearch where i to add or remove this what file ? — Almokhtar, May 06 '22 at 16:34
@Almokhtar you will find it in index settings / mapping, it’s not stored in any file. Have a look [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-custom-analyzer.html), hope it’s clarify a bit — Rob, May 07 '22 at 12:34

score 13 · Accepted Answer · edited May 20 '22 at 19:02

Credit goes to @Rob in the comments:

{
  "analysis": {
    "filter": {
      "shingle": {
        "type": "shingle",
        "min_shingle_size": 2,
        "max_shingle_size": 3
      }
    },
    "analyzer": {
      "trigram": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["shingle"]
      },
      "reverse": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["reverse"]
      }
    }
  }
}

Elasticsearch 7 The [standard] token filter has been removed

1 Answers1