I'm trying to use ElasticSearch for an application I'm building, and I am hosting it on Rackspace servers. However, the auto-discovery feature is not working. I thought that it was because auto-discovery uses broadcast and multicast to find the other nodes with the matching cluster name. I found this article saying that Rackspace does now support multicast and broadcast with their new Cloud Networks feature. Then following the article's instructions I created a network, and added that network to both of the servers the nodes were running on. I then tried restarting ElasticSearch on both nodes, but they didn't find each other, and each declared themselves as the "master" (here's the output from the logs):
[2013-04-03 22:14:03,516][INFO ][node ] [Nemesis] {0.20.6}[2752]: initializing ...
[2013-04-03 22:14:03,530][INFO ][plugins ] [Nemesis] loaded [], sites []
[2013-04-03 22:14:07,873][INFO ][node ] [Nemesis] {0.20.6}[2752]: initialized
[2013-04-03 22:14:07,873][INFO ][node ] [Nemesis] {0.20.6}[2752]: starting ...
[2013-04-03 22:14:08,052][INFO ][transport ] [Nemesis] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/166.78.177.149:9300]}
[2013-04-03 22:14:11,117][INFO ][cluster.service ] [Nemesis] new_master [Nemesis][3ih_VZsNQem5W4csDk-Ntg][inet[/166.78.177.149:9300]], reason: zen-disco-join (elected_as_master)
[2013-04-03 22:14:11,168][INFO ][discovery ] [Nemesis] elasticsearch/3ih_VZsNQem5W4csDk-Ntg
[2013-04-03 22:14:11,202][INFO ][http ] [Nemesis] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/166.78.177.149:9200]}
[2013-04-03 22:14:11,202][INFO ][node ] [Nemesis] {0.20.6}[2752]: started
[2013-04-03 22:14:11,275][INFO ][gateway ] [Nemesis] recovered [0] indices into cluster_state
The other node's log:
[2013-04-03 22:13:54,538][INFO ][node ] [Jaguar] {0.20.6}[3364]: initializing ...
[2013-04-03 22:13:54,546][INFO ][plugins ] [Jaguar] loaded [], sites []
[2013-04-03 22:13:58,825][INFO ][node ] [Jaguar] {0.20.6}[3364]: initialized
[2013-04-03 22:13:58,826][INFO ][node ] [Jaguar] {0.20.6}[3364]: starting ...
[2013-04-03 22:13:58,977][INFO ][transport ] [Jaguar] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/166.78.63.101:9300]}
[2013-04-03 22:14:02,041][INFO ][cluster.service ] [Jaguar] new_master [Jaguar][WXAO9WOoQDuYQo7Z2GeAOw][inet[/166.78.63.101:9300]], reason: zen-disco-join (elected_as_master)
[2013-04-03 22:14:02,094][INFO ][discovery ] [Jaguar] elasticsearch/WXAO9WOoQDuYQo7Z2GeAOw
[2013-04-03 22:14:02,129][INFO ][http ] [Jaguar] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/166.78.63.101:9200]}
[2013-04-03 22:14:02,129][INFO ][node ] [Jaguar] {0.20.6}[3364]: started
[2013-04-03 22:14:02,211][INFO ][gateway ] [Jaguar] recovered [0] indices into cluster_state
Is adding the network not enough (Rackspace also gave me an IP for this network)? Do I need to somehow specify in the conf file to check that network when using multicast to find other nodes?
I also found this article which offered a different approach. Per the article's instructions I put this into /config/elasticsearch.yml:
cloud:
account: account #
key: account key
compute:
type: rackspace
discovery:
type: cloud
However, then when I tried to restart ElasticSearch I got this:
Stopping ElasticSearch...
Stopped ElasticSearch.
Starting ElasticSearch...
Waiting for ElasticSearch.......
WARNING: ElasticSearch may have failed to start.
And it did fail to start. I checked into the log file for any errors, but this was all that was there:
[2013-04-03 22:31:00,788][INFO ][node ] [Chamber] {0.20.6}[4354]: initializing ...
[2013-04-03 22:31:00,797][INFO ][plugins ] [Chamber] loaded [], sites []
And it stopped there without any errors and without continuing.
Has anyone successfully gotten ElasticSearch to work in the Rackspace cloud before? I know that the unicast option is also available, but I'd prefer to not have to specify each IP address individually, as I would like it to be easy to add other nodes later. Thanks!
UPDATE
I haven't solved the issue yet, but after some searching I found this post that says the "old" cloud plugin was discontinued and replaced with just an Ec2 plugin for Amazon's cloud, which explains why the changes I made to the config file do not work.
