Elasticsearch in Docker container cluster

Question

I want to run 2 instances of Elasticsearch on 2 different hosts.

I have built my own Docker image based on Ubuntu 14.04 and the 1.3.2 version of Elasticsearch. If I run 2 ES containers on 1 host, each instance can see and communicate with the other; but when I run 2 instances of ES on 2 different hosts, it didn't work. The 9300 port of the container is bind to the 9300 host's port.

Is it possible to create an ES cluster with my configuration?

score 18 · Answer 1 · answered Mar 26 '15 at 02:02

I was able to get clustering working using unicast across two docker hosts. I just happen to be using the ehazlett/elasticsearch image, but I do not think this should matter all that much. The really important bit seems to be setting the network.publish_host setting to a public or routable IP its docker host.

Configuration

docker-host-01

eth0: 192.168.1.10
Docker version 1.4.1, build 5bc2ff8/1.4.1

docker-host-02

eth0: 192.168.1.20
Docker version 1.4.1, build 5bc2ff8/1.4.1

Building the Cluster

On Docker Host 01

docker run -d \
  -p 9200:9200 \
  -p 9300:9300 \
  ehazlett/elasticsearch \
  --cluster.name=unicast \
  --network.publish_host=192.168.1.10 \
  --discovery.zen.ping.multicast.enabled=false \
  --discovery.zen.ping.unicast.hosts=192.168.1.20 \
  --discovery.zen.ping.timeout=3s \
  --discovery.zen.minimum_master_nodes=1

On Docker Host 02

docker run -d \
  -p 9200:9200 \
  -p 9300:9300 \
  ehazlett/elasticsearch \
  --cluster.name=unicast \
  --network.publish_host=192.168.1.20 \
  --discovery.zen.ping.multicast.enabled=false \
  --discovery.zen.ping.unicast.hosts=192.168.1.10 \
  --discovery.zen.ping.timeout=3s \
  --discovery.zen.minimum_master_nodes=1

I am currently using elasticsearch:2.4.1, want to configure cassandra on two different host. Is it possible to run two nodes cluster who are not in the same LAN? — Jinna Balu, Aug 18 '17 at 15:21
i have tried running these two docker containers in two different ec2 nstances replacing the ip address. i am unable to connect each other. created one index in one machine(node) is nor reflected in another node — Jinna Balu, Aug 18 '17 at 15:35

score 18 · Answer 2 · edited Jul 08 '20 at 09:29

Using docker-compose is much easier than running it manually in command line:

elasticsearch_master:
    image: elasticsearch:latest
    command: "elasticsearch -Des.cluster.name=workagram -Des.node.master=true -Des.node.data=false"
    environment:
       - ES_HEAP_SIZE=512m
    ports:
      - "9200:9200"
      - "9300:9300"

elasticsearch1:
    image: elasticsearch:latest
    command: "elasticsearch -Des.cluster.name=workagram -Des.discovery.zen.ping.unicast.hosts=elasticsearch_master"
    links:
      - elasticsearch_master
    volumes:
      - "/opt/elasticsearch/data"
    environment:
       - ES_HEAP_SIZE=512m
elasticsearch2:
    image: elasticsearch:latest
    command: "elasticsearch -Des.cluster.name=workagram -Des.discovery.zen.ping.unicast.hosts=elasticsearch_master"
    links:
      - elasticsearch_master
    volumes:
      - "/opt/elasticsearch/data"
    environment:
       - ES_HEAP_SIZE=512m

I am learning docker and also setting up a ES cluster. Do you know where do you use the docker-compose tags like ealsticsearch_master, elasticsearch1, elasticsearch2. Where do we mention it? Stupid question may be. — animageofmine, Nov 16 '16 at 21:48
this doesn't work if we run in three different machine, and if they are not in same LAN — Jinna Balu, Aug 18 '17 at 14:53

score 3 · Answer 3 · answered Feb 20 '15 at 20:15

3

You should be able to communicate the two containers running in different hosts as far as the host machines are accessible between them in the ports needed. I think your problem is that you are trying to use ElasticSearch multicast discovery, but if then you need to expose also port 54328 of the containers. If it doesn't work you can also try to configure ElasticSearch using unicast, setting adequately the machines IP's in your elasticsearch.yml.

answered Feb 20 '15 at 20:15

Javier Cortejoso

8,851
3
26
27

thank you for your answer, I have forgot to expose the port 54328. After have expose this port it's didn't work. Configure ES to use unicast can be the solution but I realy need the multicast protocol – rival lucas Feb 23 '15 at 16:22
I haven't tried but there are sources saying multicast doesn't work in a Docker setup. But you can configure an elastic setup using unicast. Check this post to see more details: www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html – Javier Cortejoso Feb 23 '15 at 16:45