4

I want to automate kubectl and helm commands using Ansible. The target machine is configured properly, so that both works on the cli in a manual shell (e.g. kubectl get nodes or helm list). But when trying to make any API calls like get the server version

- name: List charts
  shell: kubectl version -v=8

It breaks with a Forbidden error. The verbose logging doesn't give me much more details:

fatal: [127.0.0.1]: FAILED! => changed=true 
  cmd: kubectl version -v=10
  delta: '0:00:00.072452'
  end: '2020-02-27 15:22:36.227928'
  msg: non-zero return code
  rc: 255
  start: '2020-02-27 15:22:36.155476'
  stderr: |-
    I0227 15:22:36.224517   27321 loader.go:359] Config loaded from file /home/user/.kube/config
    I0227 15:22:36.225211   27321 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.11.3 (linux/amd64) kubernetes/a452946" 'https://k8smaster01:6443/version?timeout=32s'
    I0227 15:22:36.225975   27321 round_trippers.go:405] GET https://k8smaster01:6443/version?timeout=32s  in 0 milliseconds
    I0227 15:22:36.225986   27321 round_trippers.go:411] Response Headers:
    I0227 15:22:36.226062   27321 helpers.go:219] Connection error: Get https://k8smaster01:6443/version?timeout=32s: Forbidden
    F0227 15:22:36.226080   27321 helpers.go:119] Unable to connect to the server: Forbidden
  stderr_lines: <omitted>
  stdout: 'Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}'
  stdout_lines: <omitted>

However, when sending a manual request to those API url like this

- name: Test master connection
  shell: curl -k https://k8smaster01:6443/version?timeout=32s

It works:

stderr_lines: <omitted>
  stdout: |-
    {
      "major": "1",
      "minor": "11",
      "gitVersion": "v1.11.3",
      "gitCommit": "a4529464e4629c21224b3d52edfe0ea91b072862",
      "gitTreeState": "clean",
      "buildDate": "2018-09-09T17:53:03Z",
      "goVersion": "go1.10.3",
      "compiler": "gc",
      "platform": "linux/amd64"
    }

Why API calls with kubectl doesn't work when executed with Ansible?

I'm behind a proxy server, but k8smaster01 is set in no_proxy. Ansible got it, I printed $no_proxy in the task for testing purpose.

For curl I used -k since its a self signed cert from k8s. This sould harm kubectl (which itself works when not running from Ansible). It also doesn't work when calling kubectl --insecure-skip-tls-verify=true get node with Ansible.

Lion
  • 16,606
  • 23
  • 86
  • 148
  • Did you try to force set KUBECONFIG environment variable in ansible to the kubernetes config path that you use when login through shell? – matiferrigno Feb 27 '20 at 14:44
  • kubectl works fine with ansible as through using shell or command module as k8s module. (https://docs.ansible.com/ansible/latest/modules/k8s_module.html). I have several playbooks using it. – matiferrigno Feb 27 '20 at 14:58
  • @matiferrigno I tried `kubectl version --kubeconfig=/home/user/.kube/config -v=8` (which should be equivalent to setting `KUBECONFIG` env variable), but with the same result. I found out that the behavior of Ansibles execution is the same as when I run `kubectl` with `sudo`. However, Ansible runs with `user` without `become` (which would result in `sudo`) - `whoami` gave me `user. Seems strange to me – Lion Feb 27 '20 at 15:00
  • @matiferrigno I already tried the build-in modules. The problem is that I want to install multiple third party helm charts, packed as `tgz` archives, but the `helm` module from Ansible doesn't seem to support archives (just folders or git repos). – Lion Feb 27 '20 at 15:03
  • It is true, I had a lot of troubles with the helm module and I had to work around that. Maybe this could help you. https://stackoverflow.com/questions/41482485/kubectl-behind-a-proxy – matiferrigno Feb 27 '20 at 15:16
  • Some times is NO_PROXY (uppercase) in other systems is no_proxy (lowercase). – matiferrigno Feb 27 '20 at 15:17
  • Are there any contraindications in your case to run **Ansible** on the same host, on which you run `kubectl` commands and use [k8s](https://docs.ansible.com/ansible/latest/modules/k8s_module.html) and [helm](https://docs.ansible.com/ansible/latest/modules/helm_module.html) modules rather than **shell** ? I thing it could make it more straightforward. Have you possibility to test it not from behind the proxy ? Setting `$no_proxy` env variable may not always work and have the desired effect. – mario Feb 27 '20 at 19:17
  • @matiferrigno I reset all *_proxy env variables for the task to empty strings, since I need the proxy just for internet connections, not for the internal k8s servers. Now I get `Connection error: Get https://k8smaster01:6443/api?timeout=32s: proxyconnect tcp: dial tcp: lookup None on 127.0.0.53:53: server misbehaving`. I replaced `127.0.0.53` by our internal dns server (that could resolve k8smaster01) for testing purpose in `/etc/resolv.conf` and got the same error but with the IP of our DNS instead of `127.0.0.53`. Its a proprietars MS DNS from AD. – Lion Mar 02 '20 at 13:29
  • 1
    @mario For testing purpose I run Ansible on my local PC where `kubectl` is configured. The `helm` module doesn't seem an option since I need to install helm charts by packed archive, but according to the docs of the underlying python module (the Ansible module itself doesn't cover this) it doesn't support this currently. – Lion Mar 02 '20 at 13:32

2 Answers2

7

I tried to unset the env variables from the proxy (since the proxy is required just for internet access) by setting empty environment variables:

- name: Kubectl test
  become: false
  shell: kubectl get no -v=10
  environment:
    http_proxy:
    https_proxy:
    no_proxy:
    HTTP_PROXY:
    HTTPS_PROXY:
    NO_PROXY:

This was a bad idea, since curl (which seems to be used inside kubectl) parse this to None and fail. Strangely, kubectl failed with an dns error:

skipped caching discovery info due to Get https://k8smaster01:6443/api?timeout=32s: proxyconnect tcp: dial tcp: lookup None on 127.0.0.53:53: server misbehaving

Found out that the main problem was that I set NO_PROXY=$no_proxy in /etc/environment, where no_proxy contains the hostname k8smaster01. Since /etc/environment doesn't resolve bash variables, the uppercase NO_PROXY just contains $no_proxy as string. So it was enough to replace NO_PROXY=$no_proxy by the corresponding value (e.h. NO_PROXY=k8smaster01).

It wasn't an issue before, because most applications seems to follow the Linux specification of using lowercase environment variables for proxy usage.

Lion
  • 16,606
  • 23
  • 86
  • 148
  • Hi @lion I thinking and... did you try add localhost networks on $no_proxy env var? localhost,127.0.0.1... or already have you it? – matiferrigno Mar 02 '20 at 15:17
1

On a local cluster (server listening on https://0.0.0.0:<any port>), put 0.0.0.0/8 in NO_PROXY, else kubectl (verify with kubectl cluster-info) will try to use your configured proxy.

frederic gde
  • 21
  • 4