1

I have a MySQL, Logstash, and ES setup but I need to set some fields to keyword type instead of text. I've read that it is not possible to do this in Logstash (logstash.conf) and so it needs to be done in ES. I've followed a similar question here and slightly modified it to PUT a mapping but I have got this error: "stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [es.path.data] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",

I am using docker-compose to start all the services at once under the same network, and so the mapping must be specified before logstash ports the data to ES. (Mapping can't be changed on a non-empty index).

I have seen other questions and they do seem a bit old so I wanted to ask if there is a better approach to doing this now.

My mapping.json

{
    "mappings": {
        "properties": {
            "authors": {"type": "keyword"},
            "tags": {"type": "keyword"}
        }
    }
}

Dockerfile

FROM elasticsearch:7.5.1

COPY ./docker-entrypoint.sh .

COPY ./mapping.json .

RUN mkdir /data && chown -R elasticsearch:elasticsearch /data && echo 'es.path.data: /data' >> config/elasticsearch.yml && echo 'path.data: /data' >> config/elasticsearch.yml

ADD https://raw.githubusercontent.com/vishnubob/wait-for-it/e1f115e4ca285c3c24e847c4dd4be955e0ed51c2/wait-for-it.sh /utils/wait-for-it.sh

# Copy the files you may need and your insert script

RUN ./docker-entrypoint.sh elasticsearch -p /tmp/epid & /bin/bash /utils/wait-for-it.sh -t 0 localhost:9200 -- curl -X PUT 'http://localhost:9200/cnas_publications' -d @./mapping.json; kill $(cat /tmp/epid) && wait $(cat /tmp/epid); exit 0;

Edit: I've used the docker-entrypoint.sh from the official repo here

Steve Hemmingsen
  • 383
  • 5
  • 19
  • you should make an API call the mappings endpoint with your mapping.json – Lupanoide Jan 15 '20 at 12:11
  • @Lupanoide I added some context above. I know that I need to call the API, but I need to call it before Logstash ports the data to ES. I'm using the official ES Docker image and it seems that just a simple cURL'ing in the Dockerfile isn't working – Steve Hemmingsen Jan 15 '20 at 12:45
  • 1
    if you need to put mapping with logstash you you use template instruction to logstash https://www.elastic.co/blog/logstash_lesson_elasticsearch_mapping – Lupanoide Jan 15 '20 at 13:27
  • please read here https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-manage_template template, manage_template, template_name – Lupanoide Jan 15 '20 at 13:34
  • 1
    @Lupanoide Thanks for point me in the right direction. Strangely others were saying that I should do it make a manual API call to ES rather than using Logstash – Steve Hemmingsen Jan 16 '20 at 02:56

1 Answers1

0

It seems that I was mistaken, and it is actually possible to define the mapping in Logstash. Assuming you're using the official elasticsearch image, create a ES template and make a volume with it to the logstash container.

Here's a sample of my output of my logstash.conf

output {
  stdout { codec => "rubydebug" }
  elasticsearch {
    hosts => "http://elasticsearch:9200"
    index => "test"
    template => "/logstash/mapping.json"
    template_name => "mapping"
    document_id => "%{[@metadata][_id]}"
  }
}

and don't forget to set index_patterns in your ES template.

Steve Hemmingsen
  • 383
  • 5
  • 19
  • Please note that at the moment of writing logstash elasticsearch output plugin supports only the legacy index templates. New index templates are not supported causing a `Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL...` error. Further details [here](https://discuss.elastic.co/t/logstash-elasticsearch-index-template/243339/5) and [here](https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/958) – dgiffone Jun 16 '21 at 15:22