3

Summary

We need to increase percolator performance (throughput).

Most likely approach is scaling out to multiple servers.

Questions

How to do scaling out right?

1) Would increasing number of shards in underlying index allow running more percolate requests in parallel?

2) How much memory does ElasticSearch server need if it does percolation only?

Is it better to have 2 servers with 4GB RAM or one server with 16GB RAM?

3) Would having SSD meaningfully help percolator's performance, or it is better to increase RAM and/or number of nodes?

Our current situation

We have 200,000 queries (job search alerts) in our job index. We are able to run 4 parallel queues that call percolator. Every query is able to percolate batch of 50 jobs in about 35 seconds, so we can percolate about:

4 queues * 50 jobs per batch / 35 seconds * 60 seconds in minute = 343 jobs per minute

We need more.

Our jobs index have 4 shards and we are using .percolator sitting on top of that jobs index.

Hardware: 2 processors server with 32 cores total. 32GB RAM. We allocated 8GB RAM to ElasticSearch.

When percolator is working, 4 percolation queues I mentioned above consume about 50% of CPU.

When we tried to increase number of parallel percolation queues from 4 to 6, CPU utilization jumped to 75%+. What is worse, percolator started to fail with NoShardAvailableActionException:

[2015-03-04 09:46:22,221][DEBUG][action.percolate ] [Cletus Kasady] [jobs][3] Shard multi percolate failure org.elasticsearch.action.NoShardAvailableActionException: [jobs][3] null

That error seems to suggest that we should increase number of shards and eventually add dedicated ElasticSearch server (+ later increase number of nodes).

Related: How to Optimize elasticsearch percolator index Memory Performance

Community
  • 1
  • 1
Dennis Gorelik
  • 1,234
  • 1
  • 11
  • 15

1 Answers1

4

Answers

How to do scaling out right?

Q: 1) Would increasing number of shards in underlying index allow running more percolate requests in parallel?

A: No. Sharding is only really useful when creating a cluster. Additional shards on a single instance may in fact worsen performance. In general the number of shards should equal the number of nodes for optimal performance.

Q: 2) How much memory does ElasticSearch server need if it does percolation only?

Is it better to have 2 servers with 4GB RAM or one server with 16GB RAM?

A: Percolator Indices reside entirely in memory so the answer is A LOT. It is entirely dependent on the size of your index. In my experience 200 000 searches would require a 50MB index. In memory this index would occupy around 500MB of heap memory. Therefore 4 GB RAM should be enough if this is all you're running. I would suggest more nodes in your case. However as the size of your index grows, you will need to add RAM.

Q: 3) Would having SSD meaningfully help percolator's performance, or it is better to increase RAM and/or number of nodes?

A: I doubt it. As I said before percolators reside in memory so disk performance isn't much of a bottleneck.

EDIT: Don't take my word on those memory estimates. Check out the site plugins on the main ES site. I found Big Desk particularly helpful for watching performance counters for scaling and planning purposes. This should give you more valuable info on estimating your specific requirements.

EDIT in response to comment from @DennisGorelik below:

I got those numbers purely from observation but on reflection they make sense.

  1. 200K Queries to 50MB on disk: This ratio means the average query occupies 250 bytes when serialized to disk.
  2. 50MB index to 500MB on heap: Rather than serialized objects on disk we are dealing with in memory Java objects. Think about deserializing XML (or any data format really) you generally get 10x larger in-memory objects.
richardpj
  • 413
  • 4
  • 10
  • Your advice matches with our observations in the last couple of months of experimenting with ElasticSearch cluster. We actually able to run 200K percolator queries on 2GB RAM nodes. How do 200K percolator queries transform into 50MB index and how 50MB index transforms into 500MB heap memory? – Dennis Gorelik Apr 24 '15 at 14:55
  • Thanks for the update. 10x in-memory increase is surprising, considering: "http://stackoverflow.com/questions/7146559/serialized-object-size-vs-in-memory-object-size-in-java The size in memory will be usually between half and double the serializable size." – Dennis Gorelik Apr 28 '15 at 14:36
  • We are not referring to 1-to-1 object on disk v object in memory (as in your quoted example). When parsing XML using a full DOM in memory your flat file explodes into quite a massive tree representation (roughly 10x). Likewise the elastic DSL query representation is extremely compact compared to the complex Lucene search object graph that it represents (also remember that ES has a ton of preconfigured behaviour wrt Lucene that is hidden from the user). – richardpj May 05 '15 at 07:32
  • 1
    Heads up, some of this may have changed in 5.0. Percolator queries are now stored on disk which probably had an impact on how much memory it uses. https://www.elastic.co/blog/elasticsearch-percolator-continues-to-evolve – MattM Aug 06 '19 at 18:44
  • 1
    thank you, @MattM the move from heap to disk makes sense as our EBS read throughput is so high and heap stays pretty much unused/low. any tips on how we could keep the disk read throughput under control as the number of queries grows? – asgs Jun 22 '22 at 08:27
  • 1
    @asgs This depends on your situation but what helped my query scale is doing a Filter + Percolate query, so that the possible matches are already narrowed before the remainder get percolated. I don't have a code example handy but hopefully that makes sense. In my case, this was useful for filtering out soft-deleted records (which is most of them) so that those weren't percolated at all. – MattM Jun 25 '22 at 23:01
  • @MattM I think it makes sense. we didn't add a filter on one of the important fields. we'll try and see how it goes. thank you for the idea! – asgs Jun 27 '22 at 09:47