12

I'm not sure why I'm getting an error No nodes are available that match all of the following predicates:: Insufficient cpu (1).

I don't recall setting any CPU limits. Unless this is some default?

The output of kubectl describe pod wordpress: 

Name:       wordpress-114465096-bn4rv
Namespace:  default
Node:       /
Labels:     app=wordpress
        pod-template-hash=114465096
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"wordpress-114465096","uid":"fff460df-7c4c-11e7-b3fd-42010a840026...
        kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container wordpress; cpu request for container cloudsql-proxy; cpu request for container nginx
Status:     Pending
IP:     
Controllers:    ReplicaSet/wordpress-114465096
Containers:
  wordpress:
    Image:  wordpress:latest
    Port:   
    Requests:
      cpu:  100m
    Environment:
      WORDPRESS_HOST:       localhost
      WORDPRESS_DB_USERNAME:    <set to the key 'username' in secret 'cloudsql-db-credentials'> Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ql6k8 (ro)
      /var/www/html from wordpress-persistent-storage (rw)
  cloudsql-proxy:
    Image:  gcr.io/cloudsql-docker/gce-proxy:1.09
    Port:   
    Command:
      /cloud_sql_proxy
      --dir=/cloudsql
      -instances=inspiring-tower-99712:europe-west1:wordpressdb=tcp:3306
      -credential_file=/secrets/cloudsql/credentials.json
    Requests:
      cpu:      100m
    Environment:    <none>
    Mounts:
      /cloudsql from cloudsql (rw)
      /etc/ssl/certs from ssl-certs (rw)
      /secrets/cloudsql from cloudsql-instance-credentials (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ql6k8 (ro)
  nginx:
    Image:  nginx:latest
    Port:   80/TCP
    Requests:
      cpu:      100m
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ql6k8 (ro)
Conditions:
  Type      Status
  PodScheduled  False 
Volumes:
  wordpress-persistent-storage:
    Type:   GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName: wordpress-disk
    FSType: ext4
    Partition:  0
    ReadOnly:   false
  cloudsql-instance-credentials:
    Type:   Secret (a volume populated by a Secret)
    SecretName: cloudsql-instance-credentials
    Optional:   false
  ssl-certs:
    Type:   HostPath (bare host directory volume)
    Path:   /etc/ssl/certs
  cloudsql:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium: 
  default-token-ql6k8:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-ql6k8
    Optional:   false
QoS Class:  Burstable
Node-Selectors: <none>
Tolerations:    node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
        node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  1h        22s     265 default-scheduler           Warning     FailedScheduling    No nodes are available that match all of the following predicates:: Insufficient cpu (1).

Output of kubectl describe node gke-wordpress-default-pool-91c14317-jdlj (the single node in the cluster):

Name:           gke-wordpress-default-pool-91c14317-jdlj
Role:           
Labels:         beta.kubernetes.io/arch=amd64
                        beta.kubernetes.io/fluentd-ds-ready=true
                        beta.kubernetes.io/instance-type=n1-standard-1
                        beta.kubernetes.io/os=linux
                        cloud.google.com/gke-nodepool=default-pool
                        failure-domain.beta.kubernetes.io/region=europe-west1
                        failure-domain.beta.kubernetes.io/zone=europe-west1-b
                        kubernetes.io/hostname=gke-wordpress-default-pool-91c14317-jdlj
Annotations:        node.alpha.kubernetes.io/ttl=0
                        volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:         <none>
CreationTimestamp:  Fri, 04 Aug 2017 17:44:08 +0100
Phase:          
Conditions:
Type            Status  LastHeartbeatTime           LastTransitionTime          Reason              Message
----            ------  -----------------           ------------------          ------              -------
NetworkUnavailable  False   Fri, 04 Aug 2017 17:44:35 +0100     Fri, 04 Aug 2017 17:44:35 +0100     RouteCreated            RouteController created a route
OutOfDisk       False   Tue, 08 Aug 2017 21:04:47 +0100     Fri, 04 Aug 2017 17:44:08 +0100     KubeletHasSufficientDisk    kubelet has sufficient disk space available
MemoryPressure  False   Tue, 08 Aug 2017 21:04:47 +0100     Fri, 04 Aug 2017 17:44:08 +0100     KubeletHasSufficientMemory  kubelet has sufficient memory available
DiskPressure        False   Tue, 08 Aug 2017 21:04:47 +0100     Fri, 04 Aug 2017 17:44:08 +0100     KubeletHasNoDiskPressure    kubelet has no disk pressure
Ready       True    Tue, 08 Aug 2017 21:04:47 +0100     Fri, 04 Aug 2017 17:44:39 +0100     KubeletReady            kubelet is posting ready status. AppArmor enabled
KernelDeadlock  False   Tue, 08 Aug 2017 21:03:56 +0100     Fri, 04 Aug 2017 17:43:19 +0100     KernelHasNoDeadlock         kernel has no deadlock
Addresses:      10.132.0.3,35.195.163.26,gke-wordpress-default-pool-91c14317-jdlj
Capacity:
cpu:        1
memory: 3794520Ki
pods:       110
Allocatable:
cpu:        1
memory: 3794520Ki
pods:       110
System Info:
Machine ID:         2643dae58dd36381dc5e8ebe124272bc
System UUID:            2643DAE5-8DD3-6381-DC5E-8EBE124272BC
Boot ID:            37002900-44ab-45b1-bbca-04d2b5866683
Kernel Version:     4.4.52+
OS Image:           Container-Optimized OS from Google
Operating System:       linux
Architecture:           amd64
Container Runtime Version:  docker://1.11.2
Kubelet Version:        v1.6.7
Kube-Proxy Version:     v1.6.7
PodCIDR:            10.24.0.0/24
ExternalID:         8419821342083849481
Non-terminated Pods:        (7 in total)
Namespace           Name                                CPU Requests    CPU Limits  Memory Requests Memory Limits
---------           ----                                ------------    ----------  --------------- -------------
kube-system         fluentd-gcp-v2.0-t3rzf                      100m (10%)  0 (0%)      200Mi (5%)  300Mi (8%)
kube-system         heapster-v1.3.0-3440173064-d66jq                138m (13%)  138m (13%)  301456Ki (7%)   301456Ki (7%)
kube-system         kube-dns-1829567597-n6kz6                   260m (26%)  0 (0%)      110Mi (2%)  170Mi (4%)
kube-system         kube-dns-autoscaler-2501648610-88ch6                20m (2%)    0 (0%)      10Mi (0%)   0 (0%)
kube-system         kube-proxy-gke-wordpress-default-pool-91c14317-jdlj     100m (10%)  0 (0%)      0 (0%)      0 (0%)
kube-system         kubernetes-dashboard-490794276-93cn2                100m (10%)  100m (10%)  50Mi (1%)   50Mi (1%)
kube-system         l7-default-backend-3574702981-509zt             10m (1%)    10m (1%)    20Mi (0%)   20Mi (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests    CPU Limits  Memory Requests Memory Limits
------------    ----------  --------------- -------------
728m (72%)  248m (24%)  700816Ki (18%)  854416Ki (22%)
Events:     <none>

The config file (production.yaml):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wordpress
  template:
    metadata:
      labels:
        app: wordpress
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - image: wordpress:latest
          name: wordpress
          imagePullPolicy: "Always"
          env:
            - name: WORDPRESS_HOST
              value: localhost
            - name: WORDPRESS_DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: username
          volumeMounts:
            - name: wordpress-persistent-storage
              mountPath: /var/www/html
        - image: nginx:latest
          name: nginx
          ports:
            - containerPort: 80
              name: nginx
        - image: gcr.io/cloudsql-docker/gce-proxy:1.09
          name: cloudsql-proxy
          command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                    "-instances=inspiring-tower-99712:europe-west1:wordpressdb=tcp:3306",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
            - name: ssl-certs
              mountPath: /etc/ssl/certs
            - name: cloudsql
              mountPath: /cloudsql
      volumes:
        - name: wordpress-persistent-storage
          gcePersistentDisk:
            pdName: wordpress-disk
            fsType: ext4

        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: cloudsql
          emptyDir:
Chris Stryczynski
  • 30,145
  • 48
  • 175
  • 286

3 Answers3

15

There are additional containers running besides the ones specified in the config. These are provisioned by default with Kubernetes and run in the kube-system namespace which are not shown in the default namespace.

You can view all the pods by kubectl get pods --all-namespaces.

These additional containers are taking up 72% of the CPU quota of the single node...

Hence the 3 containers at 10% cpu quota would exceed 100% of the CPU quota (because 72% + (3 * 10) > 100%)...

As to why 72% of the containers is being allocated to the other containers - the question is asked here: Why does a single node cluster only have a small percentage of the cpu quota available?

Additional resources that may be usefull: How to reduce CPU limits of kubernetes system resources?

However, I was able to get the containers to run with sufficient CPU by adding additional nodes to the cluster. In addition, the high cpu instances seem to be more efficiently allocated on Google Cloud.

Chris Stryczynski
  • 30,145
  • 48
  • 175
  • 286
0

I had got the same error but I created a cluster with just one node. So, I updated my cluster with min and max number of nodes with the following command:

gcloud container clusters update <mycluster-name> --enable-autoscaling --min-nodes=1 --max-nodes=15

And the very next moment, pod/pods changed status from Pending to Running.

typhi
  • 1
-1

The error is pretty self explanatory, you are requesting 300m (100m per container in your pod) of cpu and your node doesn't have the budget to schedule it. (You seem to have just a 1 node cluster?)

You can describe the node to see how much cpu time is already scheduled.

I'm not sure what's adding those requests to your deployment though since you aren't stating those requirements in your template. Probably a resource quota.

Brett Wagner
  • 1,134
  • 2
  • 10
  • 18
  • Yes it's a single 1 node cluster currently. Where are these quatas set though? How much 'm' does a single node have? What resources / docs should I consult? I get the error via updating my deployment config (via `kubectl apply -f production.yaml`). – Chris Stryczynski Aug 08 '17 at 18:50
  • `kubectl describe node ` will cpu on the node. I believe 1 CPU = 1000m. The expression 0.1 is equivalent to the expression 100m. Here is more details. https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ – sfgroups Aug 08 '17 at 21:33
  • I only have 3 containers in the config which I would assume means 0.3 CPU - but there is some built in containers running anyway. As to why they are allocated 70% of my node - that is another question... – Chris Stryczynski Aug 09 '17 at 08:50