2

I have started reading elasticsearch life cycle management and trying to understand how ILM works. I understood some terminologies like below

  • Indices - The actual data stores in it as "index"
  • Index pattern - To select multiple indices
  • Index Template - It is template to apply some set of "settings" and "mapping". Also used to select index policy
  • Index Alias - From here, I understood "aliases are like soft links or shortcuts to actual indexes"

Below are something confusing or I don't get it

  • Index Rollover - The index alias points to new index like

Before Rollover(Assume a policy configured )

       +--------------+
       |              |
       | metricbeat   |   +----------------+
+------>  (Aliase)    +--->metricbeat7.1   |
       |              |   |                |
       +--------------+   +----------------+

After rollover, the indices are like below(unlink the current index and point to new index)

                                 +--------------------+
                                 | metricbeat7.1      |
                                 | (read only)        |
       +--------------------+    +--------------------+
       |                    |
       |                    |
+----->+  meatricbeat       |     +---------------------+
       |   (Aliase)         |     | metricbeat-0001     |
       |                    +----->  (write index)      |
       +--------------------+     +---------------------+

Am I correct on rollover concept? In blow screen from kibana, There is option "Move to warm phase on rollover" means, move metricbeat7.1 index(from above example) in warm phase? because "rollover happned"? correct?

But what if I unselect "Move to warm phase on rollover", that is still index goes warm phase? right? why do I need that option?

enter image description here

And lastly, what is

  • leader index
  • follower index
  • "unfollow" index
  • regual index?
Pang
  • 9,564
  • 146
  • 81
  • 122
Veerendra K
  • 2,145
  • 7
  • 32
  • 61

1 Answers1

5

Your understanding of the rollover concept is correct. The main idea is that the client keeps indexing data without having to know:

  • if/when the index grows too big
  • if/when to create a new index and under which conditions
  • how to name those indexes

So what the client actually writes to is an alias that points to a single index. The ILM feature knows when it is time to rollover, i.e. when the existing index has reached the conditions that warrant the creation of a new index. When that's the case, the ILM creates a new index and switches the alias to the new index. All the while, the client keeps writing and doesn't notice anything.

It is important to know what a hot/warm architecture is. Basically, recent data is kept on (fast, powerful) hot nodes and less recent data is moved to (slower) warm nodes.

So, when an index is rolled over there are two choices: 1. either the index is kept on hot nodes for a few more days (on the UI you can select the duration before moving the index to the warm nodes) 2. or the index is moved to warm nodes immediately on rollover to make room for the latest index with the most recent data (that's what move to warm phase on rollover means)

Finally, leader/follower indexes are a different concept more related to cross-cluster replication (CCR) than ILM. A good explanation of what CCR is can be found in this blog article.

Val
  • 207,596
  • 13
  • 358
  • 360
  • yep, your answer make sense – Veerendra K May 21 '20 at 11:06
  • @Val I have a data which can be updated at any time! so i can not set the time being data in index and can not set up the Hot warm Architecture! and now I will cross more then 60GB data soon! so what should be the architecture I should follow! – Nitin Oct 01 '20 at 08:49
  • @Nitin as this question is closed, please feel free to create a new question explaining your use case and requirements. – Val Oct 01 '20 at 08:59
  • @Val i have created new question please check link https://stackoverflow.com/questions/64154032/elasticsearch-index-architecture-for-large-data-and-having-more-update-delete-op – Nitin Oct 02 '20 at 14:44