3

I have a replication controller running with the following spec:

apiVersion: v1
kind: ReplicationController
metadata:
  name: owncloud-controller
spec:
  replicas: 1
  selector:
    app: owncloud
  template:
    metadata:
      labels:
        app: owncloud
    spec:
      containers:
      - name: owncloud
        image: adimania/owncloud9-centos7
        ports:
          - containerPort: 80
        volumeMounts:
          - name: userdata
            mountPath: /var/www/html/owncloud/data
        resources:
          requests:
            cpu: 400m
      volumes:
        - name: userdata
          hostPath:
            path: /opt/data

Now I run a hpa using autoscale command.

$ kubectl autoscale rc owncloud-controller --max=5 --cpu-percent=10

I have also started heapster using kubernetes run command.

$ kubectl run heapster --image=gcr.io/google_containers/heapster:v1.0.2 --command -- /heapster --source=kubernetes:http://192.168.0.103:8080?inClusterConfig=false --sink=log

After all this, the autoscaling never kicks in. From logs, it seems that the actual CPU utilization is not getting reported.

$ kubectl describe hpa owncloud-controller
Name:               owncloud-controller
Namespace:          default
Labels:             <none>
Annotations:            <none>
CreationTimestamp:      Thu, 26 May 2016 14:24:51 +0530
Reference:          ReplicationController/owncloud-controller/scale
Target CPU utilization:     10%
Current CPU utilization:    <unset>
Min replicas:           1
Max replicas:           5
ReplicationController pods: 1 current / 1 desired
Events:
  FirstSeen LastSeen    Count   From                SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----                -------------   --------    ------          -------
  44m       8s      92  {horizontal-pod-autoscaler }            Warning     FailedGetMetrics    failed to get CPU consumption and request: metrics obtained for 0/1 of pods
  44m       8s      92  {horizontal-pod-autoscaler }            Warning     FailedComputeReplicas   failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 0/1 of pods

What am I missing here?

Aditya Patawari
  • 771
  • 1
  • 7
  • 15
  • Which version of the heapster are you running? I would also log into the node and run to check the status `docker stats`. And also you could run `curl "/api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/model/namespaces//pods//metrics/cpu/usage` – Naveen May 27 '16 at 21:05
  • @Naveen I am using image from gcr.io/google_containers/heapster:v1.0.2. Both the URLs that you showed give 404. There is only one namespace "default" in my system. I am running kubernetes 1.2.0. Heapster is actually seeing the pod, as I checked the logs of heapster pod http://pastebin.com/4qDwLGVs but somehow, that is not getting communicated to the hpa. – Aditya Patawari May 28 '16 at 19:07
  • did you resolve? I have a lot of errors in the heapster pod'd log: `E0603 15:26:22.392273 1 reflector.go:205] k8s.io/heapster/metrics/sources/kubelet/kubelet.go:339: Failed to list *api.Node: Get https://kubernetes.default/api/v1/nodes?resourceVersion=0: dial tcp: lookup kubernetes.default on 192.168.1.180:53: no such host` – DarkSkull Jun 03 '16 at 16:56

2 Answers2

1

Most probably heapster is running in a wrong namespace ("default"). HPA expects heapster to be in "kube-system" namespace. Please, add --namespace=kube-system to kubectl run heapster command.

0

I installed hepaster under the name space "kube-system" and it worked. After running heapster, make sure it's running before you use HPA for your application.

How to run Heapster with Kubernetes cluster

I put all files here https://gitlab.com/abushoeb/kubernetes/tree/master/heapster. They are collected from the official Kubernetes Repository and made minor changes.

How to run Heapster

Go to the directory heapster where you have grafana.yaml, heapster.yaml and influxdb.yaml and run following command

$ kubectl create -f  .

How to stop Heapster

Go to the same heapster directory and then run following command

$ kubectl delete -f  .

How to check Heapster is running

You can access heapster metric model from the pod where heapster is running to make sure heapster is working. It can be accessed via web browser by accessing http://heapster-pod-ip:heapster-service-port/api/v1/model/metrics/. The same result can be seen by executing following command.

$ curl -L http://heapster-pod-ip:heapster-service-port/api/v1/model/metrics/

If you see the list of metrics then heapster is running correctly. You can also browse grafana dashboard to see it (find the ip of the pod where grafana is running and the access it http://grafana-pod-ip:grafana-service-port).

Full documentation of Heapster Metric Model are available here.

Also just run ($ kubectl cluster-info) and see if it shows results like this:

Kubernetes master is running at https://cluster-ip:6443

Heapster is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/heapster

kubernetes-dashboard is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

monitoring-grafana is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana

monitoring-influxdb is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb

Check influxdb

You can also check influxdb if it has data in it. Install Influxdb Client on your local machine to get connected to infuxdb database.

$ influx -host <cluster-ip> -port <influxdb-service-port>

Some Sample influxdb queries

  • show databases
  • use db-name
  • show measurements
  • select value from "cpu/node_capacity"

Reference and Help

Tim Barrass
  • 4,813
  • 2
  • 29
  • 55
Abu Shoeb
  • 4,747
  • 2
  • 40
  • 45