How to increasing `fs.inotify.max_user_watches` for GKE Nodes

Question

I'm trying to resolve an Error “ENOSPC: System Limit for Number of File Watchers Reached” issue; which typically is solved by increasing the fs.inotify.max_user_watches value via sysctl on the Host environment. Additionally, I am not sure if part of the issue is related to using "Spot" Nodes.

Unfortunately, all my attempts to set or overwrite this value have failed. Either due to lack of permissions: e.g. /proc/sys/fs/inotify/max_user_watches: Read-only file system

And when trying to configure the GKE Nodes themselves, the linuxConfig.sysctl options does not appear to support fs.inotify.max_user_watches.

Node config: pool.yaml

kubeletConfig: {}
linuxConfig:
 sysctl:
   fs.inotify.max_user_watches: '1048576'

~ gcloud container node-pools update POOL_NAME \
    --cluster=CLUSTER_NAME \
    --system-config-from-file=pool.yaml

ERROR: (gcloud.container.node-pools.update) 
ResponseError: code=400, message=Unsupported kernel parameter fs.inotify.max_user_watches.

Any help, specific to GKE, would be greatly appreciated!

score 2 · Answer 1 · answered Jul 12 '22 at 00:50

I found this answer which uses a DaemonSet to modify all the Nodes. How to change the file-system watcher limit in Kubernetes (fs.inotify.max_user_watches)

node-setup-daemon-set.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-setup
  namespace: kube-system
  labels:
    k8s-app: node-setup
spec:
  selector:
    matchLabels:
      name: node-setup
  template:
    metadata:
      labels:
        name: node-setup
    spec:
      containers:
      - name: node-setup
        image: ubuntu
        command: ["/bin/sh","-c"]
        args: ["/script/node-setup.sh; while true; do echo Sleeping && sleep 3600; done"]
        volumeMounts:
          - name: node-setup-script
            mountPath: /script
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
      volumes:
        - name: node-setup-script
          configMap:
            name: node-setup-script
            defaultMode: 0755
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-setup-script
  namespace: kube-system
data:
  node-setup.sh: |
    #!/bin/bash
    set -e

    # change the file-watcher max-count on each node to 524288

    # insert the new value into the system config
    sysctl -w fs.inotify.max_user_watches=524288

    # check that the new value was applied
    cat /proc/sys/fs/inotify/max_user_watches

Then run

k apply -f node-setup-daemon-set.yaml

Note: Original thread mentions security concerns...

How to increasing `fs.inotify.max_user_watches` for GKE Nodes

1 Answers1