4

We have a setup, where Metricbeat is deployed as a DaemonSet on a Kubernetes cluster (specifically -- AWS EKS).

All seems to be functioning properly, but the kubelet connection.

To clarify, the following module:

- module: kubernetes
  enabled: true
  metricsets:
    - state_pod
  period: 10s
  hosts: ["kube-state-metrics.system:8080"]

works properly (the events flow into logstash/elastic).

This module configuration, however, doesn't work in any variants of hosts value (localhost/kubernetes.default/whatever):

- module: kubernetes
  period: 10s
  metricsets:
    - pod
  hosts: ["localhost:10255"]
  enabled: true
  add_metadata: true
  in_cluster: true

NOTE: using cluster IP instead of localhost (so that it goes to control plane) also works (although doesn't retrieve the needed information, of course).

The configuration above was taken directly from the Metricbeat documentation and immediately struck me as odd -- how does localhost get translated (from within Metricbeat docker) to corresponding kubelet?

The error is, as one would expect, in light of the above:

error making http request: Get http://localhost:10255/stats/summary: 
dial tcp [::1]:10255: connect: cannot assign requested address

which indicates some sort of connectivity issue.

However, when SSH-ing to any node Metricbeat is deployed on, http://localhost:10255/stats/summary provides the correct output:

{
  "node": {
   "nodeName": "...",
   "systemContainers": [
    {
     "name": "pods",
     "startTime": "2018-12-06T11:22:07Z",
     "cpu": {
      "time": "2018-12-23T06:54:06Z",
      ...
     },
     "memory": {
      "time": "2018-12-23T06:54:06Z",
      "availableBytes": 17882275840,
      ....

I must be missing something very obvious. Any suggestion would do.

NOTE: I cross-posted (and got no response for a couple of days) the same on Elasticsearch Forums

ZenMaster
  • 12,363
  • 5
  • 36
  • 59
  • Be aware that unless your DaemonSet is `hostNetwork: true`, "localhost" means the **Pod**'s localhost, not the Node's localhost. You might be happier injecting the Pod's `status.hostIP` into metricbeat and letting it contact the Node by IP, not "localhost". Is your DS `hostNetwork: true`? – mdaniel Dec 25 '18 at 06:46
  • @MatthewLDaniel It have been, at some point. The problem is, it impacts all the DNS resolutions. So, if, for example, in metricbeat setting there is a reference to logstash via DNS name of logstash.system (you know, the DNS name ..cluster.local.svc)... this gets mangled. Which means, of course, that `localhost` can't work with `logstash.namespace` at the same time. – ZenMaster Dec 25 '18 at 13:31
  • 1
    Makes sense; so I would guess two things: try injecting the Pod's Node's IP via `{"env":[{"name": "POD_IP", "valueFrom": {"fieldRef": "status.hostIP"}}]}` or you can also try pointing the container's DNS at the cluster's DNS, even in `hostNetworking: true` mode – mdaniel Dec 26 '18 at 04:16
  • @MatthewLDaniel Yep... That's what I did.... – ZenMaster Dec 27 '18 at 13:55
  • 1
    That's what you did ... and it was successful, or that's what you already tried and it didn't work? Also, which of those did you try, since there was an "or" in my suggestion – mdaniel Dec 27 '18 at 19:12
  • 1
    @MatthewLDaniel Sorry, I hadn't realized it would be ambiguous. I injected `hostIP` via downward API and left `hostNetworking` on default, which is `false`. Care to make this into an answer so I'd accept it? – ZenMaster Dec 29 '18 at 03:46

1 Answers1

6

Inject the Pod's Node's IP via the valueFrom provider in the env: list:

env:
- name: HOST_IP
  valueFrom:
    fieldRef: status.hostIP

and then update the metricbeat config file to use the host's IP:

hosts: ["${HOST_IP}:10255"]

which metricbeat will resolve via its environment variable config injection

mdaniel
  • 31,240
  • 5
  • 55
  • 58
  • 1
    I needed a nested fieldPath also, i.e. (in JSON) `{"name": "HOST_IP", "valueFrom": {"fieldRef": {"fieldPath": "status.hostIP"}}}` – mqsoh Feb 12 '19 at 17:28