I am using KubeSpray to provision a two node cluster on AWS. By default, the --kubelet-certificate-authority parameter is not used. However, I would like to set it.
I do not know the correct setting for --kubelet-certificate-authority. When I set it to /etc/kubernetes/pki/ca.crt I see messages like the following in my logs:
TLS handshake error from 10.245.207.223:59278: remote error: tls: unknown certificate authority
In order to create an isolated environment for testing, I SSH'ed to my controller node to run this command which runs apiserver on the non-standard port of 4667. I copied these values directly from /etc/kubernetes/manifests/kube-apiserver.yaml. You'll need to adjust the value to match your own cluster. I purposely am running the container in interactive mode so that I can see the log messages.
sudo docker run \
--name api-server-playground \
-it \
--rm \
--network host \
--volume /etc/kubernetes:/etc/kubernetes:ro \
--volume /etc/pki:/etc/pki:ro \
--volume /etc/ssl:/etc/ssl/:ro \
k8s.gcr.io/kube-apiserver:v1.18.9 \
kube-apiserver \
--advertise-address=10.245.207.223 \
--allow-privileged=true \
--anonymous-auth=True \
--apiserver-count=1 \
--authorization-mode=Node,RBAC \
--bind-address=0.0.0.0 \
--client-ca-file=/etc/kubernetes/ssl/ca.crt \
--cloud-config=/etc/kubernetes/cloud_config \
--cloud-provider=aws \
--enable-admission-plugins=NodeRestriction \
--enable-aggregator-routing=False \
--enable-bootstrap-token-auth=true \
--endpoint-reconciler-type=lease \
--etcd-cafile=/etc/ssl/etcd/ssl/ca.pem \
--etcd-certfile=/etc/ssl/etcd/ssl/node-ip-10-245-207-223.ec2.internal.pem \
--etcd-keyfile=/etc/ssl/etcd/ssl/node-ip-10-245-207-223.ec2.internal-key.pem \
--etcd-servers=https://10.245.207.119:2379 \
--event-ttl=1h0m0s \
--insecure-port=0 \
--kubelet-client-certificate=/etc/kubernetes/ssl/apiserver-kubelet-client.crt \
--kubelet-client-key=/etc/kubernetes/ssl/apiserver-kubelet-client.key \
--kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP \
--profiling=False \
--proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.crt \
--proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client.key \
--request-timeout=1m0s \
--requestheader-allowed-names=front-proxy-client \
--requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.crt \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--secure-port=6447 \
--service-account-key-file=/etc/kubernetes/ssl/sa.pub \
--service-cluster-ip-range=10.233.0.0/18 \
--service-node-port-range=30000-32767 \
--storage-backend=etcd3 \
--tls-cert-file=/etc/kubernetes/ssl/apiserver.crt \
--tls-private-key-file=/etc/kubernetes/ssl/apiserver.key
Now it's possible to SSH into the controller again and use curl to interact with the custom apiserver container.
- Set the APISERVER variable to point to the controller node. I'm using the value of the
advertise-addressparameter above.
APISERVER=https://10.245.207.223:6447
- Get a token.
TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}"|base64 --decode); echo "TOKEN=$TOKEN"
- Make an api request. This request will fail because "Peer's Certificate issuer is not recognized.".
curl --header "Authorization: Bearer $TOKEN" -X GET $APISERVER/api
An error message will appear in the docker log. It will look like this:
I0921 14:39:07.662368 1 log.go:172] http: TLS handshake error
from 10.245.207.223:59278: remote error: tls: unknown certificate authority
- Now use the
-kcurl parameter to bypass the recognition issue.
curl -k --header "Authorization: Bearer $TOKEN" -X GET $APISERVER/api
{
"kind": "APIVersions",
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "10.245.207.223:6447"
}
]
}
You'll see the request works correctly. However, I don't want to use the -k parameter. So I tried to use the certificate authority from the apiserver.
- Get the certificate authority from the apiserver.
echo | \
openssl s_client -connect $APISERVER 2>/dev/null | \
openssl x509 -text | \
sed -n "/BEGIN CERTIFICATE/,/END CERTIFICATE/p" \
> apiserver.ca.crt
- Use the certificate authority file for the api request.
curl --cacert apiserver.ca.crt --header "Authorization: Bearer $TOKEN" -X GET $APISERVER/api
UPDATE
Following a thought prompted by Wiktor's response, I added /etc/kubernetes/ssl/ca.crt as the certificate authority. And used that file in my curl command.
curl --cacert /etc/kubernetes/ssl/ca.crt --header "Authorization: Bearer $TOKEN" -X GET $APISERVER/api
This worked.
