Multi-Language ElasticSearch Support

Question

I am indexing messages from all around the world but mainly Thailand. The indexed messages will most likely contain either English or Thai.

Does anyone know the best way to set the ES index so that it will return good search results for both Thai and English searches?

I've tried using this setting:

{
    "settings": {
        "analysis" : {
            "analyzer" : {
                "default" : {
                    "type" : "cjk"
                }
            }
        }
    }
}

The results for the cjk analyser are not great when searching in Thai. I actually don't know why that is but any help would be very much appreciated!

score 2 · Answer 1 · answered Mar 17 '16 at 09:48

The cjk analyzer is used to generate bigrams for Chinese, Japanese and Korean but not Thai. As Thai is a non-space language this analyzer doesn't tokenize the sentence. The recommended analyzer to use for Thai language is the thai analyzer.

{
    "settings": {
        "analysis" : {
            "analyzer" : {
                "default" : {
                    "type" : "thai"
                }
            }
        }
    }
}

There is other option to analyse Thai data using the ICU Analysis Plugin that provides the icu_tokenizer. This tokenizer supports Thai, Lao, Chinese, Japanese and Korean languages. You can find the plugin by this link: ICU Analysis Plugin

After install the plugin you can use the tokenizer this way:

{
    "settings": {
        "analysis" : {
            "analyzer" : {
                "default" : {
                    "type": "custom",
                    "tokenizer": "icu_tokenizer"
                }
            }
        }
    }
}

The second method(ICU tokenizer) seems to have worked for me. Thanks! — YDF, Oct 29 '21 at 13:37

score 1 · Answer 2 · answered Aug 23 '14 at 09:13

You could implement a custom thai analyzer as described in: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-lang-analyzer.html#thai-analyzer

And, to make it a bit more useful, also add a new filter in order to use org.apache.lucene.analysis.th.ThaiWordFilterFactory from the Apache Lucene by doing like so:

curl -X PUT http://localhost:9200/test -d '{
  "settings":{
    "analysis":{
      "analyzer":{
        "default":{
          "type":"custom",
          "tokenizer":"standard",
          "filters":[ "standard","thai","lowercase", "stop", "kstem" ]
        }
      }
    },
    "filter": {
      "thai": {
        "type": "org.apache.lucene.analysis.th.ThaiWordFilterFactory"
      }
    }
  }
}’

Then, you could test with:

http://localhost:9200/test/_analyze?analyzer=thai&text=สวัสดี+hello

Hope this helps you.

Multi-Language ElasticSearch Support

2 Answers2