0

I have one opensearch cluster that receives logs from fluentd. Now I want to apply the ISM policy to rollover my index as they reach to certain threshold value.

I am following this document to create the ISM policy https://opensearch.org/docs/latest/im-plugin/ism/policies/#sample-policy-with-ism-template-for-auto-rollover.

The current configuration is something like this-

  1. The fluentd sends all the logs to specific index pattern like "mylogs-k8s-namespace", so if there are 10 namespaces, I will get 10 indices created.
logstash_format false
index_name mylogs-${record['kubernetes']['namespace_name']}
  1. Next, I created one ISM policy which includes the rollover condition-
PUT _plugins/_ism/policies/rollover_policy
{
  "policy": {
    "description": "Example rollover policy.",
    "default_state": "rollover",
    "states": [
      {
        "name": "rollover",
        "actions": [
          {
            "rollover": {
              "min_size": "10mb"
            }
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": {
      "index_patterns": ["mylogs-kube-system*"],
      "priority": 100
    }
  }
}
  1. Then I created one template which will apply this policy to all new indices.
PUT _index_template/ism_rollover
{
  "index_patterns": ["mylogs-kube-system*"],
  "template": {
   "settings": {
    "plugins.index_state_management.rollover_alias": "mylogs-kube-system"
   }
 }
}
  1. As per the documentation, the next step is to create index with above alias
PUT mylogs-kube-system-000001
{
  "aliases": {
    "mylogs-kube-system": {
      "is_write_index": true
    }
  }
}

Now here comes the problem-

If the fluentd already started pushing the logs to my index "mylogs-kube-system" then the above step (4) does not work. it gives an error that an index with the same name already exists.

This makes sense as the fluentd has already started pushing the logs and we cannot have the alias, index or streams with the same name.

To overcome from this, I have to stop my fluend, delete the index "mylogs-kube-system, in this case" and then first apply the policy and alias (step 1 to 4) and then start the fluentd again. This way it works fine and the rollover happens.

However, As I understand this is not a good solution, we cannot keep on stopping our fluentd every time a new namespace gets added. I am looking for a concreate solution to make this work.

I have tried following things but got no luck-

  1. Changing the index-name in fluentd (step 1) to logstash_prefix with date however the logs keep on getting added in new index (mylogs-kube-system-27052022) etc, but the rollover does not happen.

  2. Tried by changing the index name in fluentd to mylogs-k8s-namespace-000001 but it sends the logs only to this index forever.

The conclusion that I can draw here is that we have to keep our index name and alias name different, but doing so, the fluentd stops sending the logs to correct alias and we start facing issue in rollover.

vivek
  • 187
  • 14
  • just a heads up that this isn't an Elasticsearch problem, as ism is an opensearch implementation of our ILM, so you may want to remove that tag – warkolm May 30 '22 at 05:05
  • @MarkWalkom, this isn't but as people who know elasticsearch might know opensearch/ism as well. I think the tag could have stayed.. – Riz May 30 '22 at 14:21
  • You do not need to create an index. The index is already being created by fluentd. I think you only need to add the alias and that's it. What is your retention period for the indices? Can't you create daily indices(easily done by fluentd) and have a retention period of maybe a day or two or more? You won't need the aliases for the 'rollover'. – Riz May 30 '22 at 14:36
  • Hi @Riz, I believe, you are suggesting not to perform step no 3 (as to avoid creating the index) however, If I don't do this, I get an error "write index not applied". Kindly suggest If I misunderstood this. Also, based on the days, the policy is working fine but now I have this requirement to rollover the logs based on size hence going with rollover alias – vivek May 31 '22 at 11:36

1 Answers1

1

Try to use Data stream instead of simple Template

PUT _index_template/log
{
  "index_patterns": [
    "log-*"
  ],
  "data_stream": {}
}
Сергей Плесовских
  • 11
  • 1