54

I want to clear cache in all the pods in my Kubernetes namespace. I want to send one request to the end-point which will then send a HTTP call to all the pods in the namespace to clear cache. Currently, I can hit only one pod using Kubernetes and I do not have control over which pod would get hit.

Even though the load-balancer is set to RR, continuously hitting the pods(n number of times, where n is the total number of pods) doesn't help as some other requests can creep in.

The same issue was discussed here, but I couldn't find a solution for the implementation: https://github.com/kubernetes/kubernetes/issues/18755

I'm trying to implement the clearing cache part using Hazelcast, wherein I will store all the cache and Hazelcast automatically takes care of the cache update.

If there is an alternative approach for this problem, or a way to configure kubernetes to hit all end-points for some specific requests, sharing here would be a great help.

Vineeth Chitteti
  • 1,454
  • 2
  • 14
  • 30
  • This is a extension of what you have already thought of. When the pod receives the http call it will broadcast to a topic (on rabbitmq or similar). All pods listen to the topic and receive the event and perform tasks in response to the event – Mohit Mutha Apr 02 '18 at 14:14
  • 2
    Thanks @MohitMutha, but we are adding one more external dependency by using an external topic/queue server. – Vineeth Chitteti Apr 02 '18 at 14:22

5 Answers5

32

Provided you got kubectl in your pod and have access to the api-server, you can get all endpoint adressess and pass them to curl:

kubectl get endpoints <servicename> \
        -o jsonpath="{.subsets[*].addresses[*].ip}" | xargs curl

Alternative without kubectl in pod:

the recommended way to access the api server from a pod is by using kubectl proxy: https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod this would of course add at least the same overhead. alternatively you could directly call the REST api, you'd have to provide the token manually.

APISERVER=$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")
TOKEN=$(kubectl describe secret $(kubectl get secrets \
     | grep ^default | cut -f1 -d ' ') | grep -E '^token' | cut -f2 -d':' | tr -d " ")

if you provide the APISERVER and TOKEN variables, you don't need kubectl in your pod, this way you only need curl to access the api server and "jq" to parse the json output:

curl $APISERVER/api/v1/namespaces/default/endpoints --silent \
     --header "Authorization: Bearer $TOKEN" --insecure \
     | jq -rM ".items[].subsets[].addresses[].ip" | xargs curl

UPDATE (final version)

APISERVER usually can be set to kubernetes.default.svc and the token should be available at /var/run/secrets/kubernetes.io/serviceaccount/token in the pod, so no need to provide anything manually:

TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token); \
curl https://kubernetes.default.svc/api/v1/namespaces/default/endpoints --silent \
     --header "Authorization: Bearer $TOKEN" --insecure \
     | jq -rM ".items[].subsets[].addresses[].ip" | xargs curl

jq is available here: https://stedolan.github.io/jq/download/ (< 4 MiB, but worth it for easily parsing JSON)

Markus Dresch
  • 5,290
  • 3
  • 20
  • 40
5

UPDATE I published this article for this approach

I have had the similar situation. Here is how I resolved it (I'm using a namespace other than "default").

Setup access to cluster Using RBAC Authorization

Access to API is done by creating a ServiceAccount, assign it to the Pod and bind a Role to it.

1.Create a ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-serviceaccount
  namespace: my-namespace

2.Create a Role: in this section you need to provide the list of resources and the list of actions you'd like to have access to. Here is the example where you'd like to list the endpoints and also get the details of a specific endpoint.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-role
  namespace: my-namespace
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "list"]

3.Bind the role to the service account

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-role-binding
  namespace: my-namespace
subjects:
- kind: ServiceAccount
  name: my-serviceaccount
roleRef:
  kind: Role
  name: my-role
  apiGroup: rbac.authorization.k8s.io

4.Assign the service account to the pods in your deployment (it should be under template.spec)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  namespace: my-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
          app: my-pod
  template:
    metadata:
      labels:
        app: my-pod
    spec:
      serviceAccountName: my-serviceaccount
      containers:
      - name: my-pod
        ...

Access Clusters Using the Kubernetes API

Having all the security aspects set, you will have enough privilege to access the API within your Pod. All the required information to communicate with API Server is mounted under /var/run/secrets/kubernetes.io/serviceaccount in your Pod. You can use the following shell script (probably add it to your COMMAND or ENTRYPOINT of the Docker image).

#!/bin/bash
# Point to the internal API server hostname
API_SERVER=https://kubernetes.default.svc

# Path to ServiceAccount token
SERVICE_ACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read this Pod's namespace
NAMESPACE=$(cat ${SERVICE_ACCOUNT}/namespace)

# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICE_ACCOUNT}/token)

# Reference the internal certificate authority (CA)
CA_CERT=${SERVICE_ACCOUNT}/ca.crt

From this point forward, it is just simple REST API call. You can read these environment variables in any language of your choice and access to API.

Here is an example of listing the endpoint for your use case

# List all the endpoints in the namespace that Pod is running
curl --cacert ${CA_CERT} --header "Authorization: Bearer ${TOKEN}" -X GET \
  "${API_SERVER}/api/v1/namespaces/${NAMESPACE}/endpoints"

# List all the endpoints in the namespace that Pod is running for a deployment
curl --cacert ${CA_CERT} --header "Authorization: Bearer ${TOKEN}" -X GET \
  "${API_SERVER}/api/v1/namespaces/${NAMESPACE}/endpoints/my-deployment"

For more information on available API endpoints and how to call them, refer to API Reference.

Iraj Hedayati
  • 1,478
  • 17
  • 23
4

For those of you trying to find an alternative, I have used hazelcast as distributed event listener. Added a similar POC on github: https://github.com/vinrar/HazelcastAsEventListener

Vineeth Chitteti
  • 1,454
  • 2
  • 14
  • 30
2

I fixed this problem by using this script. You just have to write the equivalent command to make the API call. I used curl to do that.

Following is the usage of the script:

function usage {
    echo "usage: $PROGNAME [-n NAMESPACE] [-m MAX-PODS] -s SERVICE -- COMMAND"
    echo "  -s SERVICE   K8s service, i.e. a pod selector (required)"
    echo "     COMMAND   Command to execute on the pods"
    echo "  -n NAMESPACE K8s namespace (optional)"
    echo "  -m MAX-PODS  Max number of pods to run on (optional; default=all)"
    echo "  -q           Quiet mode"
    echo "  -d           Dry run (don't actually exec)"
}

For example to run command curl http://google.com on all pods of a service with name s1 and namespace n1, you need to execute ./kcdo -s s1 -n n1 -- curl http://google.com.

Lokesh
  • 2,842
  • 7
  • 32
  • 47
  • 2
    Some explanation of the script, as well as code snippets are desirable. External links disappear, and dumping a script on people without any explanation is not helpful. – Abhijit Sarkar Dec 21 '18 at 02:15
  • I was in a hurry so I didn't provide much detail. Anyway, I have updated the answer. Let me know if its ok. – Lokesh Dec 21 '18 at 05:54
  • Nice script, thanks! How can i do curl on the IPs of all pods using this script? For e.g. i want to execute /status/live on all pods using this script and not curl http://google.com – Ramandeep Singh Sep 09 '20 at 14:14
  • Got the solution: ./kcdo-s s1 -n n1 -- curl http://localhost:8080/status/live. So, localhost can be used through the script and it works. – Ramandeep Singh Sep 09 '20 at 15:12
1

I needed access to all pods so I can change log level on a class so I did from the inside of one of the pods:

// Change level to DEBUG
host <service-name>| awk '{print $4}' | while read line; do
curl --location --request POST "http://$line:9111/actuator/loggers/com.foo.MyClassName" \
--header 'Content-Type: application/json' \
--data-raw '{"configuredLevel": "DEBUG"}' 
done
// Query level on all pods
host <service-name>| awk '{print $4}' | while read line; do    
curl --location --request GET "http://$line:9111/actuator/loggers/com.foo.MyClassName"
echo
done

You need host and curl to execute it.

Not sure if this is good practice.

Chobicus
  • 2,024
  • 2
  • 17
  • 26