Troubles with docker compose + image elasticsearch:6.8.0

Question

I was trying to upgrade the version from my elastic image from 5.6 to 6.8.0, but when I run ddev start the ES container was not getting up.

DDEV logs from the elasticsearch service

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=1
[2020-02-28T21:32:29,269][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [unknown] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.0.jar:6.8.0]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.0.jar:6.8.0]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.0.jar:6.8.0]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.0.jar:6.8.0]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.0.jar:6.8.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.0.jar:6.8.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.0.jar:6.8.0]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

Additionally, I cannot use the mem_limit parameter because I'm using the latest version 3.0.

docker-compose.elasticsearch.yaml:

version: '3.6'
services:
  elasticsearch:
    container_name: ddev-${DDEV_SITENAME}-elasticsearch
    hostname: ${DDEV_SITENAME}-elasticsearch
    image: elasticsearch:6.8.0
    ports:
      - "9200"
      - "9300"
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - VIRTUAL_HOST=$DDEV_HOSTNAME
      - HTTP_EXPOSE=9200
    ulimits:
      memlock:
        soft: -1
        hard: -1
    labels:
      com.ddev.site-name: ${DDEV_SITENAME}
      com.ddev.approot: $DDEV_APPROOT
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
      - ".:/mnt/ddev_config"
  web:
    links:
      - elasticsearch:elasticsearch

volumes:
  elasticsearch:
    name: "${DDEV_SITENAME}-elasticsearch"

What am I doing wrong or what else is missing?

Update

I've tried the following solution according to an solution posted here

 version: '3.6'
services:
  elasticsearch:
    container_name: ddev-${DDEV_SITENAME}-elasticsearch
    hostname: ${DDEV_SITENAME}-elasticsearch
    image: elasticsearch:6.8.0
    ports:
      - "9200"
      - "9300"
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xmx1024m -Xms1024m"
      - VIRTUAL_HOST=$DDEV_HOSTNAME
      - HTTP_EXPOSE=9200
      - node.max_local_storage_nodes=3
    ulimits:
      memlock:
        soft: -1
        hard: -1
    labels:
      com.ddev.site-name: ${DDEV_SITENAME}
      com.ddev.approot: $DDEV_APPROOT
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
      - ".:/mnt/ddev_config"
  web:
    links:
      - elasticsearch:elasticsearch

volumes:
  elasticsearch:
    name: "${DDEV_SITENAME}-elasticsearch"

but now, throws me the following error:

Unless you use Docker Swarm there is no real need for Docker Compose 3. You could just stay on 2 and be fine. Otherwise follow https://stackoverflow.com/questions/42345235/how-to-specify-memory-cpu-limit-in-docker-compose-version-3 for the equivalent features on 3. — xeraa, Mar 01 '20 at 10:35

score 5 · Accepted Answer · answered Feb 29 '20 at 11:22

5

I just had the same issue while installing ES 7.6 on my ubuntu AWS instance, which already had another ES version installed. And if you look closer at the end of your error message it mentioned:

Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

Please refer to more details about this setting in the official ES doc.

To resolve the issue, you need to set node.max_local_storage_nodes to more than 1 in your elasticsearch.yml file.

More information about the error is available in the Elasticsearch source code in org.elasticsearch.env.NodeEnvironment#NodeEnvironment method.

if (nodeLock == null) {
    final String message = String.format(
        Locale.ROOT,
        "failed to obtain node locks, tried [%s] with lock id%s;" +
            " maybe these locations are not writable or multiple nodes were started without increasing [%s] (was [%d])?",
        Arrays.toString(environment.dataFiles()),
        maxLocalStorageNodes == 1 ? " [0]" : "s [0--" + (maxLocalStorageNodes - 1) + "]",
        MAX_LOCAL_STORAGE_NODES_SETTING.getKey(),
        maxLocalStorageNodes);
    throw new IllegalStateException(message, lastException);
}

answered Feb 29 '20 at 11:22

Amit

30,756
6
57
88

While this will solve your immediate issue, it's more fighting a symptom than finding a solution. The real question is why do you still have that lock on the data directory? Is the 5.x node still running? Did you kill the process and it couldn't remove the lock? This setting is generally not required and pretty uncommon — the default limit is there for a reason and I don't think you want to change that for your scenario. – xeraa Mar 01 '20 at 10:33
@xeraa, you are right, but this can be treated as a quick fix and based on the above questions response we can provide more info. – Amit Mar 02 '20 at 02:34
@OpsterElasticsearchNinja i've tried your solution but right now i'm facing the following trouble `ElasticsearchUncaughtExceptionHandler] [unknown] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment ` – Andres Guerrero Mar 02 '20 at 13:51
@AndresGuerrero please provide the entire stacktrace – Amit Mar 02 '20 at 14:02
@please add image to ur question and din't paste the link pls :) – Amit Mar 02 '20 at 14:09
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/208850/discussion-between-opster-elasticsearch-ninja-and-andres-guerrero). – Amit Mar 02 '20 at 14:22

score 2 · Answer 2 · answered Mar 25 '20 at 15:09

2

I had same issue, the solution that worked for me was to remove the volume that host the elasticsearch data completely. It will delete the node.lock, and the entire index, but you can recreate it again.

answered Mar 25 '20 at 15:09

Alberto G. Viu

21
1

its not always feasible to delete the data node and we shouldn't do it as you will lose the data, without taking the proper backup, btw for information, Elasticsearch latest version(7.X) changed the data folder structure, so you will not face the issue in latest version. – Amit Mar 25 '20 at 15:21
1

Deleting volumes worked for me too. Be very careful, remember, it DELETES EVERYTHING. Here you go to DELETE IMAGES `docker rmi $(docker images -a -q)` and delete VOLUMES `docker volume rm $(docker volume ls -q)` Good luck! – Under-qualified NASA Intern Nov 27 '21 at 06:45
In my case, I didn't have to delete all of the images and volumes, just the ones dedicated to Elasticsearch. – jlstrecker Jun 23 '22 at 18:30

score 0 · Answer 3 · answered Jun 26 '23 at 10:38

What worked for me was changing the volume name, mark volumes.elasticsearch6 (twice) below in docker-compose.elasticsearch.yaml:

version: '3.6'
services:
  elasticsearch:
    container_name: ddev-${DDEV_SITENAME}-elasticsearch
    hostname: ${DDEV_SITENAME}-elasticsearch
    image: elasticsearch:6.8.23
    expose:
      - "9200"
      - "9300"
    environment:
      - cluster.name=docker-cluster
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - VIRTUAL_HOST=$DDEV_HOSTNAME
      - HTTP_EXPOSE=9200:9200
      - HTTPS_EXPOSE=9201:9200
    labels:
      com.ddev.site-name: ${DDEV_SITENAME}
      com.ddev.approot: $DDEV_APPROOT
    volumes:
      - elasticsearch6:/usr/share/elasticsearch/data
      - ".:/mnt/ddev  _config"
    healthcheck:
      test: ["CMD-SHELL", "curl --fail -s localhost:9200"]

volumes:
  elasticsearch6:

Troubles with docker compose + image elasticsearch:6.8.0

3 Answers3