Elasticsearch synonyms case sensitive results

Question

Could anyone give me a tip what I'm doing wrong. I'm trying to set up elasticsearch this way

{
    "settings" : {
        "index" : {
            "analysis" : {
                "analyzer" : {
                    "synonym" : {
                        "type" : "custom",
                        "tokenizer" : "whitespace",
                        "filter" : ["en_US", "lowercase", "synonym"]
                    }
                },
                "filter" : {
                    "synonym" : {
                        "type" : "synonym",
                        "synonyms_path" : "analysis/synonym.txt"
                    },
                    "en_US" : {
                        "type" : "hunspell",
                        "locale" : "en_US",
                        "dedup" : true
                    }
                }
            }
        }
    },
    "mappings" : {
        "jdbc" : {
            "properties" : {
                "Title" : {
                    "type" : "string",
                    "search_analyzer" : "synonym",
                    "index_analyzer" : "standard"
                },
                "Abstract" : {
                    "type" : "string",
                    "search_analyzer" : "synonym",
                    "index_analyzer" : "standard"
                }
            }
        }
    }
}

My synonym.txt file contains

beer, ale, lager, cark ale
Lithuania, Republic of Lithuania, Lithuanian

So it's time to try out my analyzer:

http://localhost:9200/jdbc/_analyze?text=beer&analyzer=synonym&pretty=true

It works as expected and returns me

{
  "tokens" : [ {
    "token" : "beer",
    "start_offset" : 0,
    "end_offset" : 4,
    "type" : "SYNONYM",
    "position" : 1
  }, {
    "token" : "ale",
    "start_offset" : 0,
    "end_offset" : 4,
    "type" : "SYNONYM",
    "position" : 1
  }, {
    "token" : "lager",
    "start_offset" : 0,
    "end_offset" : 4,
    "type" : "SYNONYM",
    "position" : 1
  }, {
    "token" : "cark",
    "start_offset" : 0,
    "end_offset" : 4,
    "type" : "SYNONYM",
    "position" : 1
  }, {
    "token" : "ale",
    "start_offset" : 0,
    "end_offset" : 4,
    "type" : "SYNONYM",
    "position" : 1
  } ]
}

However, querying this:

http://localhost:9200/jdbc/_analyze?text=Lithuanian&analyzer=synonym&pretty=true

Would return only:

{
  "tokens" : [ {
    "token" : "lithuanian",
    "start_offset" : 0,
    "end_offset" : 10,
    "type" : "word",
    "position" : 1
  } ]
}

Any tips?

possible duplicate of [Case insensitivity does not work](http://stackoverflow.com/questions/17266830/case-insensitivity-does-not-work) — BMW, Nov 19 '14 at 01:34
http://stackoverflow.com/questions/22522863/elasticsearch-case-insensitive-query-string-query-with-wildcards — BMW, Nov 19 '14 at 01:34

keety · Accepted Answer · 2014-11-19T12:58:57.330

During Analysis the token filters are applied in the order specified. So in the above case "Lithuania" is first converted to lowercase "lithuania". Since the synonym file does not contain this case version of the token no synonym conversions occur. This section in elasticsearch guide talks more about this.

There are two ways you could to go about this depending on the use case either change the order of token filters in the custom analyzer to :

  "synonym" : {
         "type" : "custom",
         "tokenizer" : "whitespace",
          "filter" : ["en_US", "synonym","lowercase"]
   }

The above would still be case sensitive but the conversion would be consistent with the above synonyms.txt

Or you could make all the synonyms in analysis/synonym.txt to be lowercase and keep the same analyzer settings as specified in the question. Example:

  beer, ale, lager, cark ale
  lithuania, republic of lithuania, lithuanian

That's what I actually did. Making every synonym lowercase was the easiest and quickets solution to get things working. Thanks! — Evaldas Buinauskas, Nov 19 '14 at 18:54
Keep in mind this creates mismatches with words like CAT (medical scan) and cat (animal). — Joshua Robinson, Jun 16 '18 at 05:28

Elasticsearch synonyms case sensitive results

1 Answers1