0

I have an elastic cluster with 3x nodes(each 6x cpu, 31GB heap , 64GB RAM) collecting 25GB logs per day , but after 3x months I realized my dashboards become very slow when checking stats in past weeks , please, advice if there is an option to improve the indexes read erformance so it become faster when calculating my dashboard stats?

Thanks!

R2D2
  • 9,410
  • 2
  • 12
  • 28
  • Did you think about increasing the shards number per node – Noa Oct 01 '21 at 20:29
  • current config is 1x shard , 1x pri , 1x replica per index , will the increasing number of shards lead to search improvements and how much need to be the shards per index? – R2D2 Oct 01 '21 at 20:33

2 Answers2

1

I would suggest you try to increase the shards number when you have more shards Elasticsearch will split your data over the shards so as a result, Elastic will send multiple parallel requests to search in a smaller data stack

for Shards number you could try to split it based on your heap memory size
No matter what actual JVM heap size you have, the upper bound on the maximum shard count should be 20 shards per 1 GB of heap configured on the server.

ElasticSearch - Optimal number of Shards per node https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index https://opster.com/elasticsearch-glossary/elasticsearch-choose-number-of-shards/

Noa
  • 315
  • 1
  • 7
  • 31
  • I'm worried if increasing number of shards will add some overhead and reaching this recommended limit of 20 shards / 1GB of heap ( for 32GB heap=640 shards ) at some point ... – R2D2 Oct 01 '21 at 21:38
  • 20 is max just add 5 and try out, I would suggest that you ingest the data into the index when no user activities – Noa Oct 02 '21 at 22:19
  • Ingestion happen all time , I cant schedule in specific time slot... – R2D2 Oct 03 '21 at 07:28
  • 1
    adding more shards seems distributed better the write load ... so I accept your answer, thank you @Fateh ! – R2D2 Feb 06 '22 at 19:47
1

It seems that the amount of data that you accumulated and use for your dashboard is causing performance problems.

A straightforward option is to increase your cluster's resources but then you're bound to hit the same problem again. So you should rather rethink your data retention policy.

Chances are that you are really only interested in most recent data. You need to answer the question what "recent" means in your use case and simply discard anything older than that.

Elasticsearch has tools to automate this, look into Index Lifecycle Management.

What you probably need is to create an index template and apply a lifecycle policy to it. Elasticsearch will then handle automatic rollover of indices, eviction of old data, even migration through data tiers in hot-warm-cold architecture if you really want very long retention periods.

All this will lead to a more predictable performance of your cluster.

Michał Szajbe
  • 8,830
  • 3
  • 33
  • 39
  • the cluster is not heavily used ( not too many paralel requests ( 3-4 simultaneous sarches via the dashboards ) but there must be possibility to see stats 4-5 months back ... – R2D2 Oct 01 '21 at 21:36