2

In our Cluster setup, we have 5-10 server nodes and 10-200 clients nodes. We dynamically scale up or down number of clients nodes based on work payloads. As of now we are using S3 based discovery but wanted to know Ignite recommendation for dynamic cluster and number of nodes recommended for S3 based discovery. If I follow below Ignite documentation, I don't get much clarity which strategy would be better for my use-case i.e. max 200 nodes in cluster:

Ignite provides two implementations of the discovery mechanism intended for different usage scenarios:

  1. TCP/IP Discovery is designed and optimized for 100s of nodes.

  2. ZooKeeper Discovery that allows scaling Ignite clusters to 100s and 1000s of nodes preserving linear scalability and performance.

Doc link: https://ignite.apache.org/docs/2.9.1/clustering/clustering

Vikas Tyagi
  • 83
  • 3

1 Answers1

2

As far as it's working fine, I won't make changes to the discovery. Speaking of the S3 IpFinder, I don't think there are any recommendations, and that it has been tested with hundreds of client nodes just because it's not popular.

The quote in the question is more about server nodes. Remember, Ignite uses a ring topology configuration by default, requiring a message to travel across all the nodes. If you have a lot of nodes, it might take some time and in this case, ZookeeprDiscovery is recommended. The problem with Zookeeper is that it requires additional software and configuration.

This is assuming you are talking about Thick Clients, not Thin ones. Having that different amount of clients looks a bit suspicious though. If it's only about a short user task like making a query or sending a compute task, consider keeping a client alive for re-usage. Check if it's possible to switch to thin clients. They aren't part of the topology.

Alexandr Shapkin
  • 2,350
  • 1
  • 6
  • 10