serviceAccount Token Volume Projection - Projected Token in "path" in manifest file is incorrect

Question

I am using serviceAccount Token Volume projection as described here. This is the manifest file I'm using:

kind: Pod
apiVersion: v1
metadata:
  name: pod-acr
spec:
  containers:
  - image: yorchaksacr.azurecr.io/busybox:1.0
    name: busybox
    command: ["/bin/sh","-c","sleep 36000"]
    volumeMounts:
    - mountPath: /var/run/secrets/tokens
      name: vault-token
  serviceAccountName: pull-acr-images
  volumes:
  - name: vault-token
    projected:
      sources:
      - serviceAccountToken:
          path: vault-token
          expirationSeconds: 7200
          audience: vault

As expected, the token is mounted to the container under /var/run/secrets/tokens/vault-token:

/ # ls -la /var/run/secrets/tokens
total 4
drwxrwxrwt    3 root     root           100 Jul 24 21:35 .
drwxr-xr-x    4 root     root          4096 Jul 24 21:35 ..
drwxr-xr-x    2 root     root            60 Jul 24 21:35 ..2019_07_24_21_35_15.018111081
lrwxrwxrwx    1 root     root            31 Jul 24 21:35 ..data -> ..2019_07_24_21_35_15.018111081
lrwxrwxrwx    1 root     root            18 Jul 24 21:35 vault-token -> ..data/vault-token

Problem is if I try to authenticate to the API server using this token the API rejects the call with 401 Unauthorized:

/ # wget --header="Authorization: Bearer $(cat /var/run/secrets/tokens/vault-token)" --no-check-certificate https://10.2.1.19:6443
Connecting to 10.2.1.19:6443 (10.2.1.19:6443)
wget: server returned error: HTTP/1.1 401 Unauthorized

However if I use the default path and token where service account tokens are projected for all pods /var/run/secrets/kubernetes.io/serviceacconts/token that works:

/ # wget --header="Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --no-check-certificate https://10.2.1.19:6443
Connecting to 10.2.1.19:6443 (10.2.1.19:6443)
saving to 'index.html'
index.html           100% |************************************************************************************************************************************************************|  2738  0:00:00 ETA
'index.html' saved

If I cat both tokens I can see they are actually different:

# cat /var/run/secrets/tokens/vault-token
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJhdWQiOlsidmF1bHQiXSwiZXhwIjoxNTY0MDEzMjcwLCJpYXQiOjE1NjQwMDYwNzAsImlzcyI6Imh0dHBzOi8vMTAuMi4xLjE5Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0IiwicG9kIjp7Im5hbWUiOiJwb2QtYWNyIiwidWlkIjoiNThiNjI5YWEtZGU4Ni00YTAzLWI3YmQtMTI4ZGFiZWVkYmQ5In0sInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJwdWxsLWFjci1pbWFnZXMiLCJ1aWQiOiJlZGE0NDlmYS1iODE2LTQ0ZGMtYTFjNi0yMWJhZWUwZmVkN2YifX0sIm5iZiI6MTU2NDAwNjA3MCwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6cHVsbC1hY3ItaW1hZ2VzIn0.UjPaTgWHPwCeeh1ltBb64hv0yEjKkRxKw_BZ3PLDA3HsJK-keXN40Khp-8mNnLQ-uYIfMgW4FXwYIm0SVeQUhM4sh4rwjAYDEfEHDah9AvhEL8I65T_jhnhT10E1M7mzk1x0RFGvjZAECd1RlYM7IuXIkEfZCI_6GRVAbX3Vmk6XF0sRh2T8DZzw8kj_Z54J2gYCt2beBnn7hC9rOC9LW9J0AFEAAQQE_UJME5y4jZD6hfJMSGOouyQm70nVGytqKVsLbzyorH5pugEqrs1Z_dLx6E3Ta9kELRPvyDZgeNiS44fEYlRApn6fZawsppc1oRNoeyMqiIPRdgQekBVfTA/ #


# cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InB1bGwtYWNyLWltYWdlcy10b2tlbi1oYjU0NyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwdWxsLWFjci1pbWFnZXMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJlZGE0NDlmYS1iODE2LTQ0ZGMtYTFjNi0yMWJhZWUwZmVkN2YiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwdWxsLWFjci1pbWFnZXMifQ.nqqhZVmBUuKVi6E3L9MEn8oW1dKd-DV4c9jcVy5mXAuEMZ1WgLlaaHFF1ibnVMjEK6VUJyJhp7w08hgSmyyh-KY4BQ5oJf1jmSySvmttJxjXW-KsMpf5rHF0ZDmgaqZwbi7FvowtoTECstFBVNoszKJUn1iV5mU_6MQkEtGTNyE4KuZ9LEvPuZxiNZ5UyW3UaHXLqF63-w_xlkfa_75E-cgXqvSSGTCb6RsTuOmVyCqganx5SpIb5EU-3Mu7hUWEhSRAh3tpcPIwjS7-NkuO0ReH7Z40rPHqkIokshUUO75WM_oPq7tlu6PSCTwOK-Jw66kzi-jqKNyKvMeWJUq4WQ/ #

Does anybody have an idea why am I seeing this behavior? I would expect both tokens to work, but apparently looks like it is not the case.

Configuration of the API Server:

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.2.1.19
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --basic-auth-file=/etc/kubernetes/pki/passwordfile
    - --service-account-issuer=https://10.2.1.19
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    image: k8s.gcr.io/kube-apiserver:v1.15.1
    imagePullPolicy: IfNotPresent

Decoding those tokens with https://jwt.io/, both tokens appear to have the same subject, so I suspect it's not an authorization problem. Perhaps it's an authentication problem then? I notice that the issuer (`"iss"` key in the JWT) is different for the two tokens... — Amit Kumar Gupta, Jul 24 '19 at 23:38
... The docs section on "Service Account Token Volume Projection" that you linked mentions certain flags that need to be passed when running the API server. Looks slightly out of date, the flag names have changed slightly, probably depends on what version of K8s you're running. See flags [here](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/). Can you check the values of the appropriate flags and update your question with that info? — Amit Kumar Gupta, Jul 24 '19 at 23:39
@AmitKumarGupta Thanks for looking into this. These are the flags I am using. I agree with you this is an authorization problem since I am getting 401 - Unauthorized as a response. I am running v1.15.1, but I was experiencing the same issue with v1.13.8. The question is why is the issuer different for both tokens since they are both related to the same service account. I have edited my question with the flags enabled for the API server. — Jorge Cortes - MSFT, Jul 25 '19 at 00:52
Confusingly, a 401 Unauthorized means an authentication error (meaning that it couldn't authenticate that the subject making the request really is the subject it claims to be, in this case, via its bearer token). An authorization error would instead return a 403 Forbidden typically. See [here](https://stackoverflow.com/questions/3297048/403-forbidden-vs-401-unauthorized-http-responses). So it is indeed an authentication error... — Amit Kumar Gupta, Jul 25 '19 at 06:40
... I think the logic for authenticating tokens is more complex than the flag descriptions make it sound. I can't fully grok: https://github.com/kubernetes/kubernetes/blob/8993fbc543c18e73668793b5d5e234c0a136735c/pkg/kubeapiserver/authenticator/config.go, in particular, I can't understand why `/var/run/secrets/kubernetes.io/serviceaccount/token` works. However, I have a hunch as to why `/var/run/secrets/tokens/vault-token` doesn't work. As the flag descriptions say, the token authenticator will try to validate that the bound audiences match the value of `--api-audiences` flag... — Amit Kumar Gupta, Jul 25 '19 at 06:42
... and if that flag isn't set, then it defaults to matching against the `--service-account-issuer`. Well the problem is your token is bound to audiences `["vault"]` but the issuer is `https://10.2.1.19`. What happens if you leave out `audience: vault` from your pod spec, or, alternatively, replace it with `audience: https://10.2.1.19`? I'm assuming here that it's easier for you to change your pod spec and retry than it is to restart the kube-apiserver with different flags. — Amit Kumar Gupta, Jul 25 '19 at 06:44
Thanks for all of your help here @AmitKumarGupta. As you mentioned, replacing audience in my pod spec to match ``--service-account-issuer`` solved the problem — Jorge Cortes - MSFT, Jul 25 '19 at 16:43

score 3 · Answer 1 · answered Jul 25 '19 at 17:42

Confusingly, 401 Unauthorized indicates an authentication problem rather than an authorization problem (see here). This means the Kubernetes service account token authenticator doesn't like the token at /var/run/secrets/tokens/vault-token (let's call it the vault-token). The two tokens differ in a couple ways. Here's the decoded vault-token:

{
  "aud": [
    "vault"
  ],
  "exp": 1564013270,
  "iat": 1564006070,
  "iss": "https://10.2.1.19",
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "pod-acr",
      "uid": "58b629aa-de86-4a03-b7bd-128dabeedbd9"
    },
    "serviceaccount": {
      "name": "pull-acr-images",
      "uid": "eda449fa-b816-44dc-a1c6-21baee0fed7f"
    }
  },
  "nbf": 1564006070,
  "sub": "system:serviceaccount:default:pull-acr-images"
}

Notice the audiences (["vault"]), issuer ("https://10.2.1.19"), and subject ("system:serviceaccount:default:pull-acr-images").

Here's the default-path-token:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "default",
  "kubernetes.io/serviceaccount/secret.name": "pull-acr-images-token-hb547",
  "kubernetes.io/serviceaccount/service-account.name": "pull-acr-images",
  "kubernetes.io/serviceaccount/service-account.uid": "eda449fa-b816-44dc-a1c6-21baee0fed7f",
  "sub": "system:serviceaccount:default:pull-acr-images"
}

Same subject, but different issuer ("kubernetes/serviceaccount") and no audiences.

I'm not sure why the default-path-token has a different issuer, or why it's authenticating correctly, but your vault-token is not authenticating correctly because the audiences don't match the issuer.

More specifically, the clue is in the doc you linked here. It says that this feature working correctly depends on how you set the following flags for the kube-apiserver:

--service-account-issuer
--service-account-signing-key-file
--service-account-api-audiences

Not sure if those docs are wrong or just out of date, but since you're using v1.15.1 the flags are now called:

--service-account-issuer
--service-account-signing-key-file string
--api-audiences

The flag documentation says of the --api-audiences flag:

Identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. If the --service-account-issuer flag is configured and this flag is not, this field defaults to a single element list containing the issuer URL.

Since you don't have this flag set and you have --service-account-issuer=https://10.2.1.19, and you have audience: vault in your pod spec, your token is going to claim its bound to the vault audience, and the token authenticator is going to try to match this against the value of the --service-account-issuer flag, and clearly those don't match.

You can make these match by specifying audience: https://10.2.1.19 in your pod spec instead of audience: vault. One caveat: this solution may technically work to ensure the token authenticates, but I'm not sure what the truly right answer is insofar as using this flags and fields in the pod spec as they're truly intended, and just making these strings match could be a bit hacky.

serviceAccount Token Volume Projection - Projected Token in "path" in manifest file is incorrect

1 Answers1