Elasticsearch application with docker compose runnable on multi-node swarm

Question

I have a small app with a python backend where I'm streaming and classifying tweets in real-time. I use elasticsearch to collect classified tweets and Kibana to make visualizations based on es data. In my frontend, I just use kibana visualizations.

For the moment, I'm trying to run my application in a multi-node swarm as a services stack but I'm having problems with my compose file.

I tried to start with elastisearch and to use this info https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html but didn't help, and I didn'd succed to deploy my docker-compose file even with just elasticsearch serivce.

This is my yml file:

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - 'ES_JAVA_OPTS=-Xms512m -Xmx512m'
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - '9200:9200'
  kibana:
    image: docker.elastic.co/kibana/kibana:7.6.2
    ports:
      - '5601:5601'

what error you are getting, also I see you mentioned `es01,es02,es03` but there is only `es01` mentioned in your docker-compose.yml — Amit, Apr 18 '20 at 13:05
so you are just trying to run a `simple single node ES` and that itself itself is not working for you? if yes, then my this SO answer might help you https://stackoverflow.com/questions/60421343/elasticsearch-docker-container-in-non-prod-mode-to-eliminate-vm-max-map-count-26/60426167#60426167 — Amit, Apr 18 '20 at 14:16
I removed es02 and eso3. Anyway, the deploy is creating the network and the es service but after I checked the state with `docker stack ps app` I saw that there is an error with non-zero exit (78). — Costi Peticilă, Apr 18 '20 at 14:20
I searched in logs and i found this: Native controller process has stopped - no new native processes can be started — Costi Peticilă, Apr 18 '20 at 14:22
can you try with `-e "discovery.type=single-node"` also ? also please update ques with latest compose file — Amit, Apr 18 '20 at 14:23
my goal is to use with multi-node but for the moment yes, I'm trying with just a single node — Costi Peticilă, Apr 18 '20 at 14:25
yeah lets go step by step, as you are running a single node, you need to add `discovery.type=single-node` to make sure elastic bootstrap understand its for development purpose and doesn't check the production settings — Amit, Apr 18 '20 at 14:27
Added proper docker-compose file, removed `kibana` as mostly you dont need it. please try with my compose file and let me know if it works — Amit, Apr 18 '20 at 14:35
It's my first time when I use docker! After I didn't succeed running on a swarm with `"docker stack deploy -c docker-compose.yml testapp` command, I tried to run on a single node with this command `docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.6.2` and I checked and ES was up but I couldn't to run also kibana (https://www.elastic.co/guide/en/kibana/current/docker.html ) — Costi Peticilă, Apr 18 '20 at 14:39
so now your es service is up ? also you mentioned in your question ```I didn'd succed to deploy my docker-compose file even with just elasticsearch service.``` ? hence I was suggesting to remove kibana and do you really need kibana this time? — Amit, Apr 18 '20 at 14:45
Yes, now it's working! For the moment I don't need kibana but i repeat, my goal is to run my application as a services stack. Thank you a lot! — Costi Peticilă, Apr 18 '20 at 16:22

score 0 · Accepted Answer · answered Apr 18 '20 at 14:34

Below is the docker-compose file which works for a single node in a development environment, which have disabled security and has discovery.type=single-node param to make sure elasticsearch production bootstrap checks are not kicked in.

version: '2.2'

services:
  #Elasticsearch Docker Images: https://www.docker.elastic.co/
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
    container_name: elasticsearch
    environment:
      - xpack.security.enabled=false 
      - discovery.type=single-node
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    cap_add:
      - IPC_LOCK
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9300:9300

volumes:
  elasticsearch-data:
    driver: local
networks:
    elastic:
        external: true

Elasticsearch application with docker compose runnable on multi-node swarm

1 Answers1