1

i am running an EFK-Stack (elastic, fluent-bit, kibana) on an azure kubernetes service. For some reasen i dont get the index lifecyle management to work properly. I added the Logstash_Format On to the output-elasticsearch.conf in order the create a new index every day, like logstash-* This is what my output-elasticsearch.conf looks like: ```

    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
        HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWD}
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False

In order to get ilm to work i followed the documentation provided by elastic (https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html). I added the following configuration according to the documentation:

  1. add index lifecycle policy:
PUT _ilm/policy/logstash_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "5mb",
            "max_age": "1h"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "2h",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
  1. Create an index template
PUT _index_template/desc_template
{
  "index_patterns": ["logstash-*"],                 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logstash_policy",
      "index.lifecycle.rollover_alias": "logstash-delete"    
    }
  }
}
  1. Bootstrap the initial index logstash with a write index alias
PUT logstash-000001
{
  "aliases": {
    "logstash": {
      "is_write_index": true
    }
  }
}

After reaching the 5mb border size, i set in the index policy, the new index logstash-000001 gets created properly. The only problem is, that the new rollover index remains empty. All the shiped logs from fluentbit gets still written to the daily index logstash-*. Am i missing something here. I also dont see any ilm config for fluent-bit available. Any help would very appreciated. Cheers Martin

Martin Hering
  • 11
  • 3
  • what index does fluentbit write to? – warkolm Aug 16 '21 at 23:19
  • Hi Mark, fluentbit writes to ```logstash-*```. Every day there is a new index ```logstash-``` generated. – Martin Hering Aug 17 '21 at 07:06
  • ok that's not compatible with ILM, you need to be writing to `logstash-delete` as it's the alias you defined in the policy – warkolm Aug 23 '21 at 03:08
  • @MarkWalkom, sorry to start a new thread after so long.. I was going thru the same situation and was able to get the new logs in my rolled over indices after specifying the index name as logstash-delete in fluentbit.. However, I am facing issue where fluentbit/fluentd creates the index automatically as soon as logs start flowing and then during bootstarap, we receive the error as index with the same name already exists. issue reference -- https://stackoverflow.com/questions/72403952/issue-with-opensearch-index-pattern-alias-and-fluentd-index-name would like to know your view on it. – vivek May 27 '22 at 11:36

0 Answers0