Elasticsearch: Max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Question

I have an issue with a systemd config for ElasticSearch.

[Unit]
Description=platform-elasticsearch
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
User={{ app_user }}
Group={{ app_group }}
Environment=ES_PATH_CONF=/platform/opt/elasticsearch-{{ elasticsearch.version }}/config
Environment=JAVA_HOME=/platform/opt/jdk{{ jdk.major_version }}_{{ jdk.minor_version }}
LimitAS=infinity
LimitRSS=infinity
LimitCORE=infinity
LimitNOFILE=100000
LimitMEMLOCK=100000
StandardOutput=syslog
StandardError=syslog
WorkingDirectory=/platform/var/app/elasticsearch
ExecStart=/platform/opt/elasticsearch-{{ elasticsearch.version }}/bin/elasticsearch
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s -TERM $MAINPID
TimeoutStopSec=60
# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143 0
Type=simple
Restart=on-failure
RestartSec=10
PIDFile=/platform/var/run/elasticsearch.pid

[Install]
WantedBy=multi-user.target

This does not seem to let me configure the vm.max_map_count setting.

Jul 20 14:53:46 scratchpad elasticsearch: [2018-07-20T14:53:46,359][INFO ][o.e.b.BootstrapChecks    ] [1oQJNUK] bound or publishing to a non-loopback     address, enforcing bootstrap checks
Jul 20 14:53:46 scratchpad elasticsearch: ERROR: [1] bootstrap checks failed
Jul 20 14:53:46 scratchpad elasticsearch: [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Jul 20 14:53:46 scratchpad elasticsearch: [2018-07-20T14:53:46,376][INFO ][o.e.n.Node               ] [1oQJNUK] stopping ...
Jul 20 14:53:46 scratchpad elasticsearch: [2018-07-20T14:53:46,414][INFO ][o.e.n.Node               ] [1oQJNUK] stopped
Jul 20 14:53:46 scratchpad elasticsearch: [2018-07-20T14:53:46,414][INFO ][o.e.n.Node               ] [1oQJNUK] closing ...
Jul 20 14:53:46 scratchpad elasticsearch: [2018-07-20T14:53:46,445][INFO ][o.e.n.Node               ] [1oQJNUK] closed
Jul 20 14:53:46 scratchpad systemd: platform-elasticsearch.service: main process exited, code=exited, status=78/n/a

The specific issue is the following:

Jul 20 14:53:46 scratchpad elasticsearch: [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

I have been able to start elastic search on the commandline with the following:

sudo su -c 'echo 262144 > "/proc/sys/vm/max_map_count"' && \ 
export JAVA_HOME=/platform/opt/jdk1.8.0_181 && \
export ES_PATH_CONF=/platform/opt/elasticsearch-6.3.1/config && \
/platform/opt/elasticsearch-6.3.1/bin/elasticsearch

can anyone tell me why LimitMEMLOCK=100000 does not work, and how I can effectively set max_map_count from within systemd.

I have also tried to set the following:

cat /etc/security/limits.d/30_elastic_limits.conf

vagrant       hard    nofile     500000
vagrant       hard    memlock     262144

but this seems to be totally ignored by systemd.

score 335 · Answer 1 · edited Apr 12 '19 at 09:21

335

Vivek's answer

sysctl -w vm.max_map_count=262144

is correct, however, the setting will only last for the duration of the session. If the host reboots, the setting will be reset to the original value.

If you want to set this permanently, you need to edit /etc/sysctl.conf and set vm.max_map_count to 262144.

When the host reboots, you can verify that the setting is still correct by running sysctl vm.max_map_count

edited Apr 12 '19 at 09:21

Raedwald

46,613
43
151
237

answered Jul 20 '18 at 18:52

Val

207,596
13
358
360

48

Do not forget to reboot or: sysctl --system – hybaken Sep 25 '18 at 08:52
2

note that if you restart the machine that configuration will be gone, so this method is not permanent. – Mustafa Salih ASLIM May 05 '21 at 07:06
1

This answer, and variations on it, seems to be the most popular answer, but what I don't see is WHERE this command should be executed. I'm trying this in Windows with WSL, and I get the error "sysctl: cannot stat /proc/sys/vm/max_map_count: No such file or directory", as if this argument is being treated like a file or command path, not a parameter being set to a value. I've also tried editing /etc/sysctl.conf in the WSL environment, and using a .wslconfig file in my Windows home directory with `kernelCommandLine = "sysctl.vm.max_map_count=262144"`. – kshetline Nov 01 '22 at 19:41
1

@kshetline this is a [Linux-only configuration](https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html) – Val Nov 02 '22 at 05:05
@Val,mI know it's a Linux config, but it's still needed for _some_ Linux environment on Windows that runs under WSL. The question is how to invoke the right Linux environment in which to make this change. – kshetline Nov 02 '22 at 14:35
1

@kshetline does this help: https://stackoverflow.com/a/63882309/4604579 ? – Val Nov 02 '22 at 14:50
@Val, yes, it was the `wsl -d docker-desktop` that I needed. For some reason that didn't work the first time I tried it, but it does now. Thanks. – kshetline Nov 02 '22 at 15:07

score 186 · Answer 2 · edited May 25 '22 at 10:43

186

For MS Windows users, using wsl subsystem

Open powershell, run

wsl -d docker-desktop

then

sysctl -w vm.max_map_count=262144

edited May 25 '22 at 10:43

Carl

2,896
2
32
50

answered Mar 09 '21 at 13:26

Hamza AZIZ

2,582
1
9
18

14

This worked for me on Windows 10 machine. Last step would be to enter `exit` in order to kick out of wsl. – user1653042 Oct 06 '21 at 18:51
1

worked just fine, thank you very much! pitty I had to copy the command to power shell – thevoyager Dec 02 '21 at 18:21
2

This doesn't seem to be a "sticky" change in Docker Desktop -- you have to run the command every time you restart Docker, which is really obnoxious. – Mr. T Dec 03 '21 at 15:39
Thanks! Worked as expected – Ytzvan Mastino Jan 07 '22 at 18:34
9

To persist the change, you can add `vm.max_map_count = 262144` to `/etc/sysctl.conf` – MonkeyDreamzzz Apr 20 '22 at 09:31
This solution works. It is also the official solution from Elastic. Thank you! – Gravity API Apr 25 '22 at 06:38
1

Persisting the change in `/etc/sysctl.conf` doesn't seem to work long-term, but https://stackoverflow.com/a/71743053/9913 does. – Jedidja Aug 19 '22 at 12:07
that's great but how to make change in docker-compose? – littleAlien Aug 05 '23 at 15:50
@littleAlien its a host setting, not a container setting. – Gunnar Aug 23 '23 at 12:53

burtsevyg · Answer 3 · 2021-03-11T19:39:20.387

113

Insert the new entry into the /etc/sysctl.conf file with the required parameter:

vm.max_map_count = 262144

it make changes permanent.

Also run:

sysctl -w vm.max_map_count=262144

change current state of kernel.

If you use docker to take effect you should restart it:

systemctl restart docker

edited Mar 11 '21 at 19:39

answered Dec 10 '19 at 12:43

burtsevyg

3,851
2
29
44

if you are not root user you should make sure you have sudo right and use sudo before execute above. – xiaojueguan Jan 08 '21 at 06:52
3

if you use docker-compose to run containers on host, you can also run this command `sysctl -w vm.max_map_count=262144` with root privilege to solve this problem. – xiaojueguan Feb 23 '21 at 02:29
2

Use `sysctl vm.max_map_count=262144` (iwithout -w) and this way the change is available only till shutdown of OS. It is a dev machine. So, i did not want to change permanently. – nashter Jun 22 '21 at 15:59
this also works for Kubernetes if anyone is wondering, just run int on all of your nodes, maybe cordon and drain each first though. – MikeSchem Jul 21 '22 at 20:19

score 54 · Answer 4 · edited Apr 12 '19 at 09:24

54

See the Elasticsearch documentation about virtual memory. On Centos you can do with following command:

sysctl -w vm.max_map_count=262144

edited Apr 12 '19 at 09:24

Raedwald

46,613
43
151
237

answered Jul 20 '18 at 17:54

Vivek Pakmode

1,016
2
10
22

score 28 · Answer 5 · answered Jul 05 '21 at 09:33

You can also resolve the memory constraint issue by using single-node discovery type. Set this in the environment: discovery.type=single-node

docker-compose.yml

services:
  es:
    image: elasticsearch
    environment:
      - discovery.type=single-node

See also:

https://github.com/docker/for-win/issues/5202#issuecomment-636028670

score 26 · Answer 6 · edited Jul 06 '19 at 15:04

This is not an answer per se but a clarification/shortcut for anyone having the op's problem from a docker container perspective. I had this problem from an application running in a docker container. And as explained here by nishant

You don’t need to increase the virtual memory for Elasticsearch at the container level, you can do it for the host machine by running this command:

sudo sysctl -w vm.max_map_count=262144

and then restart your docker-containers.

As explained by val above setting this max_map_count this way won't persist upon the restart of the machine on which is running the docker container. and so you will need to save it in a more persistent manner as explained by him above.

score 20 · Answer 7 · edited Nov 04 '21 at 15:27

I ran into the same issue for the single node elastic search cluster. As perelastic-search documentation, for running single node, you have use "discovery.type=single-node" in the docker run command.

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.13.3

In docker-compose.yml, you can specif below:

version: '3.1'
services:
 elastic_search:
  image: docker.elastic.co/elasticsearch/elasticsearch:7.13.3
  container_name: es01
  environment:
   - discovery.type=single-node
   - node.name=es01
   - bootstrap.memory_lock=true
   - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
  ports:
   - 9200:9200
  networks:
   - elastic
networks:
 elastic:

score 17 · Answer 8 · answered Apr 04 '22 at 19:52

17

For WSL, you can create config file (.wslconfig) at:

C:\Users\[your user]\.wslconfig

and put in it:

[wsl2]
kernelCommandLine = "sysctl.vm.max_map_count=262144"

So that you do not have to set max_map_count every time you restart PC

answered Apr 04 '22 at 19:52

topolm

385
7
13

score 10 · Answer 9 · answered Nov 30 '19 at 18:41

Please run the following command: sysctl -w vm.max_map_count=262144 to increase the default virtual memory used by the Elasticsearch.

Note: When you run the above-mentioned command your problem will get resolved but, this will be a temporary solution as node/system/container gets restart your changes will go. So if you want to set this permanently, you need to edit /etc/sysctl.conf and set vm.max_map_count to 262144.

For more detail click here.

score 5 · Answer 10 · edited Jul 30 '22 at 18:08

5

This worked for me.

 sysctl -w vm.max_map_count=262144

edited Jul 30 '22 at 18:08

JAMSHAID

1,258
9
32

answered Jun 04 '22 at 11:38

Raw_Wish

151
1
6

score 3 · Answer 11 · answered May 10 '23 at 14:16

3

According to this answer, you can run:

sudo sysctl -w vm.max_map_count=262144

Just run the following command to see the value changed:

sysctl vm.max_map_count

answered May 10 '23 at 14:16

A M

831
1
14
26

score 2 · Answer 12 · edited Sep 21 '20 at 06:13

2

Max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144].
Please run the command below to fix this issue:

sysctl -w vm.max_map_count=262144

edited Sep 21 '20 at 06:13

סטנלי גרונן

2,917
23
46
68

answered Sep 21 '20 at 04:43

deepu kumar singh

319
2
6

score 2 · Answer 13 · answered Nov 12 '21 at 05:55

2

vm.max_map_count=262144

Add this line in to below path

vim /etc/sysctl.conf

or go to this link https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html

answered Nov 12 '21 at 05:55

Abdul Aziz Khan

29
1

1

What is the additional insight which your post provides in addition to existing answer? Especially compared to the most upvoted answer and the one it refers to. Try to avoid the impression that you create answers by using info from other answers to the same question, especially when even the summary is already there... – Yunnosch Nov 12 '21 at 06:28
As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Nov 12 '21 at 06:28

score 1 · Answer 14 · answered Aug 30 '22 at 14:48

1

You might also set node.store.allow_mmap: false.

However, this setting has memory implications for production workloads as mentioned here. It helped me to deploy Elasticsearch 8.4.0 in a minikube kubernetes cluster using the elastic operator.

answered Aug 30 '22 at 14:48

aemaem

935
10
20

score 0 · Answer 15 · answered Apr 04 '22 at 05:15

In case anyone is on elasticsearch 8.0 with kubernetes ECK operator.

Set runAsUser: 0

elastic/cloud-on-k8s#5410 (comment)

Starting with 8.0 Elasticsearch container is not running as root anymore, which is required to run sysctl. You can add runAsUser: 0 to force the init container to run as root

score 0 · Answer 16 · edited Feb 07 '23 at 14:05

if you are using yml file

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: prod
spec:
  version: 8.6.0
  nodeSets:
    - name: master-nodes
      count: 3
      config:
        node.roles: ["master"]
      podTemplate:
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                privileged: true
                runAsUser: 0
              command: ["sh", "-c", "sysctl -w vm.max_map_count=262144"]
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi
            storageClassName: standard
    - name: data-nodes
      count: 10
      config:
        node.roles: ["data"]
      podTemplate:
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                privileged: true
                runAsUser: 0
              command: ["sh", "-c", "sysctl -w vm.max_map_count=262144"]
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 100Gi
            storageClassName: standard

walkman · Answer 17 · 2022-12-16T06:06:35.680

-1

this will make changing now, and always work even if you reboot the device

echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.conf;sudo sysctl -w vm.max_map_count=262144

edited Dec 16 '22 at 06:06

answered Dec 09 '22 at 11:23

walkman

1,743
2
10
10

[A code-only answer is not high quality](//meta.stackoverflow.com/questions/392712/explaining-entirely-code-based-answers). While this code may be useful, you can improve it by saying why it works, how it works, when it should be used, and what its limitations are. Please [edit] your answer to include explanation and link to relevant documentation. – Stephen Ostermiller Dec 10 '22 at 10:30

Elasticsearch: Max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

17 Answers17

Linked