4

I'm using pm2 to watch the directory holding the source code for my app-server's NodeJS program, running within a Kubernetes cluster.

However, I am getting this error:

ENOSPC: System limit for number of file watchers reached

I searched on that error, and found this answer: https://stackoverflow.com/a/55763478

# insert the new value into the system config
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

However, I tried running that in a pod on the target k8s node, and it says the command sudo was not found. If I remove the sudo, I get this error:

sysctl: setting key "fs.inotify.max_user_watches": Read-only file system

How can I modify the file-system watcher limit from the 8192 found on my Kubernetes node, to a higher value such as 524288?

Venryx
  • 15,624
  • 10
  • 70
  • 96

2 Answers2

3

I found a solution: use a privileged Daemon Set that runs on each node in the cluster, which has the ability to modify the fs.inotify.max_user_watches variable.

Add the following to a node-setup-daemon-set.yaml file, included in your Kubernetes cluster:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-setup
  namespace: kube-system
  labels:
    k8s-app: node-setup
spec:
  selector:
    matchLabels:
      name: node-setup
  template:
    metadata:
      labels:
        name: node-setup
    spec:
      containers:
      - name: node-setup
        image: ubuntu
        command: ["/bin/sh","-c"]
        args: ["/script/node-setup.sh; while true; do echo Sleeping && sleep 3600; done"]
        env:
          - name: PARTITION_NUMBER
            valueFrom:
              configMapKeyRef:
                name: node-setup-config
                key: partition_number
        volumeMounts:
          - name: node-setup-script
            mountPath: /script
          - name: dev
            mountPath: /dev
          - name: etc-lvm
            mountPath: /etc/lvm
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
      volumes:
        - name: node-setup-script
          configMap:
            name: node-setup-script
            defaultMode: 0755
        - name: dev
          hostPath:
            path: /dev
        - name: etc-lvm
          hostPath:
            path: /etc/lvm
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-setup-config
  namespace: kube-system
data:
  partition_number: "3"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-setup-script
  namespace: kube-system
data:
  node-setup.sh: |
    #!/bin/bash
    set -e

    # change the file-watcher max-count on each node to 524288

    # insert the new value into the system config
    sysctl -w fs.inotify.max_user_watches=524288

    # check that the new value was applied
    cat /proc/sys/fs/inotify/max_user_watches

Note: The file above could probably be simplified quite a bit. (I was basing it on this guide, and left in a lot of stuff that's probably not necessary for simply running the sysctl command.) If others succeed in trimming it further, while confirming that it still works, feel free to make/suggest those edits to my answer.

Venryx
  • 15,624
  • 10
  • 70
  • 96
1

You do not want to run your container as a privileged container if you can help it.

The solution here is to set the following kernel parameters, then restart your container(s). The container(s) will use the variables from the kernel that your container is running within. This is because containers do not run separate kernels on Linux hosts (containers use the same kernel).

fs.inotify.max_user_watches=10485760
fs.aio-max-nr=10485760
fs.file-max=10485760
kernel.pid_max=10485760
kernel.threads-max=10485760

You should paste the above into: /etc/sysctl.conf.

Steven
  • 11
  • 1