1

I'm using docker/elk image to display my data in kibana dashboard (Version 6.6.0) and It works pretty good. I started the service like using below command.

Docker Image git repo:

 https://github.com/caas/docker-elk

Command:

    sudo docker-compose up --detach

Expecting that it will run background, and did as expected. After two days the server up and running the and third day the kibana alone getting stopped. and Used below command to make it up and running.

    sudo docker run -d <Docer_image_name>

It's up and running when I use docker ps command. But when I tried to hit the kibana server in chrome browser it says not reachable.

So I just used to below command to restart the service.

    sudo docker-compose down

After that I can see kibana server in chrome browser which is up and running but I do see all my data is lost.

I used below URL in jenkins to collect the data.

`http://hostname:9200/ecdpipe_builds/extern`al

Any idea how can I resolve this issue?

ArrchanaMohan
  • 2,314
  • 4
  • 36
  • 84
  • I guess you are using dockerized database without data persisting mechanism. In that case the database is initialized every-time it starts and looses it's data when it stops due to non-persistent data. You can use volume to persist database file. – Rafaf Tahsin Oct 10 '19 at 05:50

2 Answers2

2

I did not see the persistent storage configuration the image you mentioned in their GitHub docker-compose file.

This is common to lost data in case of docker container if you did not provide persistent storage configuration. so docker-compose down may cause to lost you data if there is no persistent configuration docker-compose file.

Persisting log data

In order to keep log data across container restarts, this image mounts /var/lib/elasticsearch — which is the directory that Elasticsearch stores its data in — as a volume.

You may however want to use a dedicated data volume to persist this log data, for instance to facilitate back-up and restore operations.

One way to do this is to mount a Docker named volume using docker's -v option, as in:

$ sudo docker run -p 5601:5601 -p 9200:9200  -p 5044:5044 \
    -v elk-data:/var/lib/elasticsearch --name elk sebp/elk

This command mounts the named volume elk-data to /var/lib/elasticsearch (and automatically creates the volume if it doesn't exist; you could also pre-create it manually using docker volume create elk-data).

So you can set these paths in your docker-compose file accordingly. Here is the link that you can check elk-docker-persisting-log-data

Adiii
  • 54,482
  • 7
  • 145
  • 148
  • Many thanks for yours response. Let me check and get back to you. – ArrchanaMohan Oct 11 '19 at 05:23
  • I'm just using the below image. Can you please let me know where I should add the volume? https://github.com/caas/docker-elk. Updated the question – ArrchanaMohan Oct 11 '19 at 06:03
  • this seems old, I will suggest using the offical one https://elk-docker.readthedocs.io/#persisting-log-data – Adiii Oct 11 '19 at 06:14
  • Looks like its been added volumes: - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro – ArrchanaMohan Oct 11 '19 at 06:14
  • I already used it but I do have some issues in that latest versioning that you mentioned. I have nested json file which is not able to handled so I don't have an option to use it so I downgraded to this. Can you pls help me how can I solve that volume issue using this build? – ArrchanaMohan Oct 11 '19 at 06:16
  • I used the latest and downgraded to old because of this issue. https://stackoverflow.com/questions/58121067/objects-in-array-is-not-well-supported-error-observed-for-elk-docker-image/58124174?noredirect=1#comment102643888_58124174 – ArrchanaMohan Oct 11 '19 at 06:18
  • try to run `sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 \ -v /home/user/elk-data:/var/lib/elasticsearch --name elk sebp/elk` and verify is there any files in `/home/user/elk-data`? once you got data here then we can mount in docker-compose – Adiii Oct 11 '19 at 06:19
1

Use docker volume or file location as persistant space

Saurav Solanki
  • 199
  • 15