Starting 2 hazelcast instances synchronously on a single machine (with hazelcast.initial.min.cluster.size=2)

Question

How can I configure Hazelcast (optimally the version I currently use: 3.1.2) to run 2 hazelcast instances on a single machine, and block the first instance during startup until both instances are present?

hazelcast.initial.min.cluster.size

The blocking behavior described above can be implemented with the help of hazelcast.initial.min.cluster.size when running both instances on different machines: With the configuration:

Config cfg = new Config(); 
cfg.setProperty("hazelcast.initial.min.cluster.size",
  Integer.toString(minimumInitialMembersInHazelCluster));
cfg.getGroupConfig().setName(clusterName);
hazelInst = Hazelcast.newHazelcastInstance(cfg);

run on different machines, I get the output

Apr 15, 2014 9:31:39 AM com.hazelcast.instance.DefaultAddressPicker
INFO: Prefer IPv4 stack is true.
Apr 15, 2014 9:31:39 AM com.hazelcast.instance.DefaultAddressPicker
INFO: Picked Address[192.168.31.105]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
Apr 15, 2014 9:31:40 AM com.hazelcast.system
INFO: [192.168.31.105]:5701 [dev] Hazelcast Community Edition 3.1.2 (20131120) starting at Address[192.168.31.105]:5701
Apr 15, 2014 9:31:40 AM com.hazelcast.system
INFO: [192.168.31.105]:5701 [dev] Copyright (C) 2008-2013 Hazelcast.com
Apr 15, 2014 9:31:40 AM com.hazelcast.instance.Node
INFO: [192.168.31.105]:5701 [dev] Creating MulticastJoiner
Apr 15, 2014 9:31:40 AM com.hazelcast.core.LifecycleService
INFO: [192.168.31.105]:5701 [dev] Address[192.168.31.105]:5701 is STARTING
Apr 15, 2014 9:31:43 AM com.hazelcast.cluster.MulticastJoiner
INFO: [192.168.31.105]:5701 [dev] 


Members [1] {
    Member [192.168.31.105]:5701 this
}

Apr 15, 2014 9:31:43 AM com.hazelcast.core.LifecycleService
INFO: [192.168.31.105]:5701 [dev] Address[192.168.31.105]:5701 is STARTED
Apr 15, 2014 9:31:43 AM HazelMultiInstanceExp <init>
SEVERE: debug: joined via JoinConfig{multicastConfig=MulticastConfig [enabled=true, multicastGroup=224.2.2.3, multicastPort=54327, multicastTimeToLive=32, multicastTimeoutSeconds=2, trustedInterfaces=[]], tcpIpConfig=TcpIpConfig [enabled=false, connectionTimeoutSeconds=5, members=[], requiredMember=null], awsConfig=AwsConfig{enabled=false, region='us-east-1', securityGroupName='null', tagKey='null', tagValue='null', hostHeader='ec2.amazonaws.com', connectionTimeoutSeconds=5}} with 1 members.
Apr 15, 2014 9:31:43 AM com.hazelcast.instance.DefaultAddressPicker
INFO: Prefer IPv4 stack is true.
Apr 15, 2014 9:31:43 AM com.hazelcast.instance.DefaultAddressPicker
INFO: Picked Address[192.168.31.105]:5702, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5702], bind any local is true
Apr 15, 2014 9:31:43 AM com.hazelcast.system
INFO: [192.168.31.105]:5702 [dev] Hazelcast Community Edition 3.1.2 (20131120) starting at Address[192.168.31.105]:5702
Apr 15, 2014 9:31:43 AM com.hazelcast.system
INFO: [192.168.31.105]:5702 [dev] Copyright (C) 2008-2013 Hazelcast.com
Apr 15, 2014 9:31:43 AM com.hazelcast.instance.Node
INFO: [192.168.31.105]:5702 [dev] Creating MulticastJoiner
Apr 15, 2014 9:31:43 AM com.hazelcast.core.LifecycleService
INFO: [192.168.31.105]:5702 [dev] Address[192.168.31.105]:5702 is STARTING
Apr 15, 2014 9:31:43 AM com.hazelcast.nio.SocketConnector
INFO: [192.168.31.105]:5702 [dev] Connecting to /192.168.31.105:5701, timeout: 0, bind-any: true
Apr 15, 2014 9:31:43 AM com.hazelcast.nio.SocketAcceptor
INFO: [192.168.31.105]:5701 [dev] Accepting socket connection from /192.168.31.105:60803
Apr 15, 2014 9:31:43 AM com.hazelcast.nio.TcpIpConnectionManager
INFO: [192.168.31.105]:5702 [dev] 60803 accepted socket connection from /192.168.31.105:5701
Apr 15, 2014 9:31:43 AM com.hazelcast.nio.TcpIpConnectionManager
INFO: [192.168.31.105]:5701 [dev] 5701 accepted socket connection from /192.168.31.105:60803
Apr 15, 2014 9:31:49 AM com.hazelcast.cluster.ClusterService
INFO: [192.168.31.105]:5702 [dev] 

Members [2] {
    Member [192.168.31.105]:5701
    Member [192.168.31.105]:5702 this
}

but run with two instances on one machine, I get

Apr 15, 2014 9:25:14 AM com.hazelcast.instance.DefaultAddressPicker
INFO: Prefer IPv4 stack is true.
Apr 15, 2014 9:25:14 AM com.hazelcast.instance.DefaultAddressPicker
INFO: Picked Address[192.168.31.105]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
Apr 15, 2014 9:25:15 AM com.hazelcast.system
INFO: [192.168.31.105]:5701 [test3] Hazelcast Community Edition 3.1.2 (20131120) starting at Address[192.168.31.105]:5701
Apr 15, 2014 9:25:15 AM com.hazelcast.system
INFO: [192.168.31.105]:5701 [test3] Copyright (C) 2008-2013 Hazelcast.com
Apr 15, 2014 9:25:15 AM com.hazelcast.instance.Node
INFO: [192.168.31.105]:5701 [test3] Creating MulticastJoiner
Apr 15, 2014 9:25:15 AM com.hazelcast.core.LifecycleService
INFO: [192.168.31.105]:5701 [test3] Address[192.168.31.105]:5701 is STARTING
Apr 15, 2014 9:25:18 AM com.hazelcast.cluster.MulticastJoiner
INFO: [192.168.31.105]:5701 [test3] 


Members [1] {
    Member [192.168.31.105]:5701 this
}

Apr 15, 2014 9:25:18 AM com.hazelcast.instance.HazelcastInstanceImpl
INFO: [192.168.31.105]:5701 [test3] HazelcastInstance waiting for cluster size of 2

with the last INFO message repeated ad infinitum.

So the cluster size is the number of nodes in the cluster, not the number of hazelcast instances?

Blocking behavior on a single machine

I use cfg.setProperty("hazelcast.initial.min.cluster.size","2") so the two distributed instances start my distributed algorithm synchronously. Furthermore, it solves the problem that some hazelcast members are not found, see https://stackoverflow.com/a/20716919/750378.

So how do I avoid both problems when running on a single machine? It would be great if the cluster size would simple be the number of running hazelcast instances. Then I could keep my configuration above no matter how I deploy my two instances (on 1 or 2 machines).

Update

I have posted an issue about the cluster size at https://github.com/hazelcast/hazelcast/issues/2292.

score 5 · Answer 1 · edited Apr 12 '14 at 10:38

5

You could use the same machine to run multiples Hazelcast instance. Under the hoods, Hazelcast binds differents ports for each instance.

Following, a simple test with two instances:

HazelcastInstance firstInstance = Hazelcast.newHazelcastInstance(new Config());
HazelcastInstance secondInstance = Hazelcast.newHazelcastInstance(new Config());

// Introducing a sample data in a map using the first instance
firstInstance.getMap("TEST_MAP").put("key", "test_value");

// Prints the value using the second instance. The result will be 'test_value'
System.out.println(secondInstance.getMap("TEST_MAP").get("key"));

When this code starts the console output is:

Members [2] {
    Member [192.168.216.1]:5702
    Member [192.168.216.1]:5703 this
}

As you can see, two instances were registered in the same IP but different ports.

edited Apr 12 '14 at 10:38

DaveFar

7,078
4
50
90

answered Apr 10 '14 at 13:37

efernandez

274
1
5

Thanks efernandez for your answer (+1). But your solution works no longer when `hazelcast.initial.min.cluster.size` 2 is used. I rephrased my question for clarity. – DaveFar Apr 15 '14 at 08:04
in my cast i have 2 instanes in log eventhough only one newHazelastInstance() i in the code. seems it is default fo 3.4.4 until you hange it in confg by hazelcast.initial.min.cluster.size prop – ses Aug 08 '15 at 18:20
Its creating as 2 different instance but when i clear cache in one instance is affecting other instance cache. I used IMap – Senthil Arumugam SP Mar 07 '17 at 15:58

score 2 · Answer 2 · answered Apr 15 '14 at 09:09

You should start instances in different threads. See following test:

public static void main(String[] args) throws InterruptedException {
        final Config cfg = new Config();
        cfg.setProperty(GroupProperties.PROP_INITIAL_MIN_CLUSTER_SIZE, "2");

        Runnable runnable = new Runnable() {
            public void run() {
                final HazelcastInstance instance = Hazelcast.newHazelcastInstance(cfg);
                IMap<Object, Object> map = instance.getMap("map");
                for (int i = 0; i < 100; i++) {
                    map.put(i, i);
                    System.out.println("put:" + i);
                }
            }
        };
        Thread thread1 = new Thread(runnable);
        Thread thread2 = new Thread(runnable);
        thread1.start();
        Thread.sleep(1000);
        thread2.start();
    }

In my actual program, I start two different JVMs, and one hazelcast instance per JVM, but that does not solve the problem. — DaveFar, Apr 15 '14 at 11:08

score 2 · Answer 3 · answered Apr 22 '14 at 00:33

2

Please see this thread for more information. Based on the info there, your best bet is to:

make sure you're using a configuration which allows for two machines in the cluster, on the same pc (check out the sample config file attached to that thread)
Run both machines on the same JVM (rather than firing up two separate JVMs)
Are running a fairly recent version (3.1.2 should be fine)

Starting two JVMs on the same physical machine might work sometimes if you make sure they don't have conflicting port settings, but you'll likely run into various problems (ie, i bet they won't be able to properly listen to the same multicast port). I'd advise against that approach.

answered Apr 22 '14 at 00:33

blueberryfields

45,910
28
89
168

Arg, I missed the bounty, sorry (was on holiday). I guess this comment will lead to the solution. I will start another bounty later for compensation. – DaveFar Apr 25 '14 at 08:13
I need to use different JVMs since my program using hazelcast is peer to peer. Do you have any advice on that, i.e. how to avoid conflicting port settings and discovery problems? I thaught the simple configuration I am using could handle all that (find an open IP port and use discovery via TCP/IP). – DaveFar Apr 25 '14 at 08:15
I don't think that will work - all of the instances on the same cluster share one port for coordination - and they won't both be able to bind to the same port when using multiple jvms. You might want to set up a vm, or setup a second loopback adapter and use a proxy (say, fiddler) to forward ports around – blueberryfields Apr 25 '14 at 12:42
(to be clear, the multicast port is what is being shared between the two vms; your setup might work if you disable muticast for the single-machine scenario) – blueberryfields Apr 25 '14 at 13:05

Starting 2 hazelcast instances synchronously on a single machine (with hazelcast.initial.min.cluster.size=2)

hazelcast.initial.min.cluster.size

Blocking behavior on a single machine

Update

3 Answers3