1

I am struggeling to find a suitable way to handle ilm rollover on dynamic indices from a logstash pipeline to elasticsearch.

The pipeline looks like that:

input {
  pipeline { 
    address => "some_pipeline_name"
  }
}

output {
  elasticsearch {
    hosts => [ "serv1", "server2" ]
    index => "%{[label1]}-%{[label2]}-%{[label3]}"
    user => "logstash_user_to_send_elasticsearch"
    password => "password1337"
  }
}

If I have an index template for the index that was dynamically created everythings works well and there are no problems with ilm and rollover.

But since I don't know all values the 3 labels can get, I don't know how to handle the ilm aspect. It is not possible to define ilm_rollover_alias dynamically and I am not able to create pipelines for every possible value of label* in advance.

How do you guys handle this problem?

Regards

Sebastian

sbstnmrwld
  • 36
  • 5
  • I'd use [data streams](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html) instead since they handle all the index creation and ILM in a transparent way, without having to bootstrap anything except an index template and its attached ILM policy. – Val Mar 17 '22 at 07:09
  • we are using datastremas as well I have the same problem here: I have no index template for unknown label values, so I cannot set ilm and activate datastream in advance Or do I miss something? – sbstnmrwld Mar 17 '22 at 07:13
  • What I would do is to add a little prefix to your index name `"xyz-%{[label1]}-%{[label2]}-%{[label3]}"` then you can build your index template pattern with `xyz-*` – Val Mar 17 '22 at 07:16
  • I tried this before and the following problem occured: There was no problem with creating the datastream or apply the template or the ILM policy, but there was no rollover since all (sub)-indices have the same rollover alias – sbstnmrwld Mar 17 '22 at 07:19
  • I think you should provide some concrete examples of labels and what error you encountered otherwise we're shooting in the dark. – Val Mar 17 '22 at 07:27

0 Answers0